Python glob.glob(dir) Memory Error

Question

I'm dealing with a memory issue when searching a folder with milions of files. Does anyone know how to overcome this situation? Is there some way to put a limit of files that glob will search? So it could be executed in chunks?

Traceback (most recent call last):
  File "./lb2_lmanager", line 533, in <module>
   main(sys.argv[1:])
 File "./lb2_lmanager", line 318, in main
    matched = match_files(policy.directory, policy.file_patterns)
  File "./lb2_lmanager", line 32, in wrapper
    res = func(*args, **kwargs)
  File "./lb2_lmanager", line 380, in match_files
    listing = glob.glob(directory)
  File "/usr/lib/python2.6/glob.py", line 16, in glob
    return list(iglob(pathname))
  File "/usr/lib/python2.6/glob.py", line 43, in iglob
    yield os.path.join(dirname, name)
  File "/usr/lib/python2.6/posixpath.py", line 70, in join
   path += '/' + b
MemoryError

Have you tried iglob rather than glob? – LexyStardust Oct 19 '15 at 14:29 — LexyStardust, Oct 19 '15 at 14:29

score 3 · Accepted Answer · edited May 23 '17 at 10:27

3

Try using generators instead of lists.
To understand what generators are read this

import glob
dir_list = glob.iglob(YOUR_DIRECTORY)
for file in dir_list:
    print file

Change YOUR_DIRECTORY to the directory that you would like to list.

edited May 23 '17 at 10:27

Community

1
1

answered Oct 19 '15 at 14:43

Guy Goldenberg

809
7
20

I'll try iglob instead of glob. – Bernardo Vale Oct 19 '15 at 14:50

Python glob.glob(dir) Memory Error

1 Answers1