1

I'm setting up a script and I'm trying to figure out why I'm forced to wait 60 seconds for the loading of my pages. I've developed a platform where I display some information about a file (its image and its details).

At the moment, I display the data of 10 folders containing around 10000 files each.

I'm using the Flask framework and FileAdmin to load my files. I've tried to analyze the time of each function in my code and I've obtained this:

all_the_files = glob.glob(directory + "/*.*") >>>>>>>> around 0.01s

for file in all_the_files:
   if '_ex' in str(file):
     try:
       filename = str(file).split('/')[-1]
       file_part1 = filename.split('_')[0]
       file_part2 = filename.split('_')[1]

>>>>>> around 1e-05s for each

found_file = [file for file in all_the_files if re.match("^"+filename+".*", str(file).split('/')[-1])]
image_path = [file for file in found_file if re.match("^"+filename+".*"+"\.png", str(file).split('/')[-1])][0]
txt_path = [file for file in found_file if re.match("^"+filename+".*"+"\.txt", str(file).split('/')[-1])][0]

>>>>> around 0.02s for each file

With these time records, I know that the third function is too long for each file.

How can I manage to reduce the loading time of my pages?

tomjn
  • 5,100
  • 1
  • 9
  • 24
Maitresage
  • 81
  • 1
  • 10
  • I've already used a SSD – Maitresage Sep 02 '19 at 09:41
  • don't you think its normal for 3 loops? – Jainil Patel Sep 02 '19 at 09:43
  • The first loop is relative to the whole folder, but the next are just relative to the first one which contains maximum 3 strings. I'm forced to loop 3 times, I didn't find another way to realize that. – Maitresage Sep 02 '19 at 09:45
  • try using fie,file1,file2 .... etc instead of just one variable file – Jainil Patel Sep 02 '19 at 09:48
  • 1
    and even post regex, so that others can make it simpler. – Jainil Patel Sep 02 '19 at 09:48
  • I'll update my question. – Maitresage Sep 02 '19 at 09:50
  • One possible optimisation: replace `str(file).split('/')[-1]` with `str(file).rpartition('/')[-1]`. Also, I don't think you need to call `str` on the filename each time - the output of `glob.glob` is a list of strings. A higher level optimisation, if your code cannot be made fast enough, might be to cache the computed data in memory and render your page using that data unless a the content of the directories is modified. – snakecharmerb Sep 03 '19 at 06:13

1 Answers1

0

It looks like you are trying to find files whose names match specific patterns. If so, os.walk() might be faster. See Find a file in python for an example similar to what you are doing.

mti2935
  • 11,465
  • 3
  • 29
  • 33