Check if all files are there in directory before proceeding to execute the remaining code?

Question

I'm trying to make sure that all the files(here in my case 'ccf_table','bis_file') are there in the directory before the code proceed to start the computations. And if either of them(bis_file,ccf_table) is not there in directory for any of the input file('filename' in my case) then just simply pass that input file.

path = "/home/Desktop/s1d_data"
for filename in os.listdir(path):
  if filename.endswith("_s1d_A.fits"):
  s1d_hdu = fits.open(filename)
  s1d_header = s1d_hdu[0].header
  date = s1d_header['DATE-OBS']
  date2 = date = date[0:19]
  ccf_table = glob('HARPS.' + date2 + '*_ccf_G2_A.tbl')
  bis_file = glob('HARPS.' + date2 + '*_bis_G2_A.fits')

  filelist = ['ccf_table','bis_file']
  while True:
    list1 = []
    for file in filelist:
      list1.append(os.path.isfile(file))

      if all(list1):
         break
      else:
         pass

  df=pd.read_table(ccf_table[0],skiprows=2,usecols=(0,4),names=['order','rv'],)
  df = df.rv.mean()
  out_name = s1d_header['OBJECT'] +'-' + s1d_header['DATE-OBS'] +'.fits'

That's what i've tried until now but didn't get the desired result.Any help would be appreciated.

Have you checked out using `os.path.isfile`? https://stackoverflow.com/questions/2259382/pythonic-way-to-check-if-a-file-exists — mooglinux, Aug 12 '18 at 23:13
It looks like that you are trying to create a workflow. If you use more of that kind of rules I would suggest to look in workflow frameworks like snakemake or luigi — nauer, Aug 13 '18 at 06:45
Isn't this basically a duplicate of your previous question or the one before that? https://stackoverflow.com/questions/51749269/how-to-skip-a-file-if-its-not-there-in-the-directory it seems like you're still struggling with getting the file processing logic of the same code right. Maybe there's someone local to you who can help? Or if you'd like to move this to a chat room I can help you with your code. — Iguananaut, Aug 15 '18 at 10:13
Also, when posting Python code please make sure to get the indentation exactly as you have it (you can copy and paste your code directly into the SO editor, select it with the mouse, and press Ctrl-K to indent it all simultaneously) — Iguananaut, Aug 15 '18 at 10:16
One other tip that might help you if you're struggling: try to use pdb, the Python debugger. If you google it you can find some great tutorials for beginners. For starters instead of running your script like `python myscript.py`, run `python -m pdb myscript.py`. This will drop you into an interactive `(pdb) ` prompt which will let you step through your code line by line and examine the values of variables at each step. Try doing this a few times and it will help you better understand what your code is doing and to think clearly and logically about it, instead of treating it as a black box. — Iguananaut, Aug 15 '18 at 10:23
Two things to ask yourself as you step through this code: What type of object (a list? a string?) is returned by the `glob()` function? What are the values in the variable `filelist`? Are they the values of the `ccf_table` and `bis_file` variables that you assigned the results of the `glob()` calls to, or something else ? Also, why is there a `while True` loop? Do you *really* need that? — Iguananaut, Aug 15 '18 at 10:26

score 0 · Answer 1 · answered Aug 16 '18 at 14:57

Well, thanks for the help everyone but i managed to solve this problem on my own by using an another approach. So i thought it's better to share how i solved it.

for filename in os.listdir("/home/Desktop/2d_spectra"):
   if filename.endswith("_e2ds_A.fits"):
     e2ds_hdu = fits.open(filename,ignore_missing_end=True)
     e2ds_header = e2ds_hdu[0].header
     blaze_file = e2ds_header['HIERARCH ESO DRS BLAZE FILE']
     date = e2ds_header['DATE-OBS']
     date2 = date = date[0:19]
     blaze_file1 = glob(blaze_file)
     bis_file = glob('HARPS.' + date2 + '*_bis_G2_A.fits')
     ccf_table = glob('HARPS.' + date2 + '*_ccf_G2_A.tbl')

     ll = np.size([blaze_file1]) + np.size(bis_file) + np.size(ccf_table)   #[] will convert the string to numpy array
        if ll == 3:      
        #did what i wanted to do.

        else:
           none

Check if all files are there in directory before proceeding to execute the remaining code?

1 Answers1