-2

I've written a function whose purpose is to open, read, access data values within the each file in an indicated directory and iteratively append each files data entries into empty lists which will then be used to construct a summary csv file.

The issue is that my lists are empty at the termination of the for loop. How do i get around this issue?

I've tried looking through existing questions and haven't found it helpful.

Here's my code:

fileid   = [] * len(os.listdir(path))   
objname  = [] * len(os.listdir(path))
obsvtype = [] * len(os.listdir(path))
filtr    = [] * len(os.listdir(path))



# Open, read, and iterate over files ending with the .fits extension in inputted path.
# Assign fits header key data to a variable and then append iteratively to respective empty list.

for fitsfile in os.listdir(path):
    if fitsfile[-5:] == '.fits':
        try:
            hdulist     = fits.open(str.format(fitsfile))

            filename    = hdulist[0].data['FILENAME']
            fileid.append(filename)


            object_name = hdulist[0].data['OBJECT']
            objname.append(object_name)

            obsvtyp     = hdulist[0].data['OBSTYPE']
            obsvtype.append(obsvtyp)

            filt        = hdulist[0].data['FILTERS']
            filtr.append(filt)
        except:
            pass

print(fileid,objname,obsvtype,filtr)


# Create an empty pandas dataframe object (data table) and assign populated data lists to dataframe.
# columns.

dataframe = pd.DataFrame(data = None)
dataframe['Filename']         = fileid     
dataframe['Object_Name']      = objname    
dataframe['Observation_type'] = obsvtype   
dataframe['Filter']           = filtr      

print(dataframe)

# Export compiled dataframe object to a .csv file in specified directory path.

dataframe.to_csv(path + 'reduc_tbl_result.csv',columns = ['Filename','Object_name',
    'Observation_Type','Filter'],index = None)

and the result:

([], [], [], []) Empty DataFrame Columns: [Filename, Object_Name, Observation_type, Filter] Index: [])

Thanks in advance.

Jean-Paul Ventura
  • 31
  • 1
  • 6
  • Can you create a Minimal, Complete and Verifiable example? https://stackoverflow.com/help/mcve – christopherlovell Jun 28 '17 at 21:18
  • Missing None in list initialiser. You're then appending to the pre initialised array. Apart from that, there must be something going awry in file reading, which I can't debug from this example – christopherlovell Jun 28 '17 at 21:54
  • First up, stop using `try: ... except: pass`. You're telling Python to shut up whenever anything goes wrong, and then coming to us when something went wrong and Python didn't tell you. **Let Python tell you when things go wrong.** – user2357112 Jun 30 '17 at 21:51
  • Fair point. I've tried that already and the result is the same... but also First up... I didn't come to you, I came to stack overflow and **Y.O.U.** volunteered your submission... – Jean-Paul Ventura Jun 30 '17 at 22:34

3 Answers3

0

In answer to your original question, you need to initialise your lists. The append is failing, but the error is caught by the try block.

fileid = []

If you have a lot of files then appending isn't very efficient, as the array must be copied in full with each append. Best to initialise the list in advance.

fileid = [None] * len(os.listdir(path))

for i, fitsfile in enumerate(os.listdir(path)):
  fileid[i] = ...

You might also want to look at using the glob module for listing all files in a directory with a given extension, see here for an example.

christopherlovell
  • 3,800
  • 4
  • 19
  • 26
  • It didn't to work. I'm still exporting an empty csv file. Any other thoughts? Thanks for the help. i've been bargaining with my code for the last couple of days. – Jean-Paul Ventura Jun 28 '17 at 21:14
  • It works for a minimal example I mocked up, so the error is possibly elsewhere in your code. See my comment above – christopherlovell Jun 28 '17 at 21:25
  • Also you say in the question the list is empty at termination, but are now saying the csv file is empty. Are the lists still empty? – christopherlovell Jun 28 '17 at 21:35
  • Yes both the lists and the .csv file are empty at completion. I have print statements written for the lists before any attempts at exporting are done and they show the lists as empty. – Jean-Paul Ventura Jun 30 '17 at 21:47
0

The following basic code can be used to append files to a list:

import os
file_list = []
for f in os.listdir(os.getcwd()):
    file_list.append(f)
print(file_list)

For your code, you want to use the path provided rather than the current working directory.

Try something simpler (ie, no processing of fits files) and see if that works and build up in complexity. It's possible that your error may not be in appending to lists, but in using fits or the pandas Dataframe.

David Rodriguez
  • 41
  • 5
  • Thanks... after some time coding my attention gets fuzzy, but you're right. I worked along that strategy and came to the appropriate conclusions. I tried the code analogous to what you posted and the files were printed correctly leading me to realize that the issue wasn't necessarily the for-loop syntax, fits.open() just wasn't recognizing the string element output of os.listdir(path) as something it could operate on. I posted my working code below if you should need it for the future/curiosity. – Jean-Paul Ventura Jul 05 '17 at 23:27
0

I've since recognized the source of my issue for those who'll need a solution in the future. My code from above was printing empty lists because the fits.open() call was not operating on string elements of the os.listdir(path) via the code below:

    for fitsfile in os.listdir(path):
        if fitsfile[-5:] == '.fits':
           try:
               hdulist     = fits.open(str.format(fitsfile))

I tried this and it made the difference:

    for fitsfile in os.listdir(path):
        if fitsfile[-5:] == '.fits':
            with fits.open(os.path.join(path,fitsfile)) as hdulist:
                try:  
                   hdu = hdulist[0]
Jean-Paul Ventura
  • 31
  • 1
  • 6