3

I am using Python 2.7.10 And pyfits 3.3. Earlier, I have used the following code to merge two tables. However, now I am getting some errors

t1 = pyfits.open(table1)[1].columns
t2 = pyfits.open(table2)[1].columns
new_columns = t1 + t2
hdu = pyfits.BinTableHDU.from_columns(new_columns)
hdu.writeto(outtable)

The error is:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/vvikraman/anaconda2/lib/python2.7/site-packages/pyfits/hdu/table.py", line 116, in from_columns
    data = FITS_rec.from_columns(coldefs, nrows=nrows, fill=fill)
 File "/home/vvikraman/anaconda2/lib/python2.7/site-packages/pyfits/fitsrec.py", line 315, in from_columns
    if arr.hdu.data is None:
ReferenceError: weakly-referenced object no longer exists
Remi Guan
  • 21,506
  • 17
  • 64
  • 87
vinu
  • 457
  • 4
  • 11

2 Answers2

4

Is there a reason you cannot use astropy (i.e. astropy.io.fits)?

In that case the idiom would be:

from astropy.table import Table, hstack
t1 = Table.read(table1)
t2 = Table.read(table2)
new = hstack([t1, t2])
new.write(outtable)

In both the read and write calls, you need to provide format='fits' if the table name extension(s) do not imply that it is FITS.

Tom Aldcroft
  • 2,339
  • 1
  • 14
  • 16
3

It's a bit confusing, and I'm not 100% sure about the cause, but here's my guess:

The result of opening the file itself, pyfits.open(...), isn't assigned to anything, because you immediately access the columns of the second HDU. That gives you a weak reference: essentially, the actual data isn't available anymore, because the file is, in a sense (and possibly real, I haven't checked) closed.
Pyfits does this to save memory.

The confusing part is that, after you've assigned t1 and t2, you can still print their definition, so it seems the actual columns are around. In fact, it's the column definitions that are still around: the actual data can't be retrieved. Hence, when you tried to create a new binary HDU from the combination t1 and t2, where you need the actual data, things fail.

The solution is to make sure you keep a proper reference to the data. There might be multiple ways to do this; this seems to work for me:

hdulist1 = pyfits.open(table1)
hdulist2 = pyfits.open(table2)
t1 = hdulist1[1].columns
t2 = hdulist2[1].columns
new_columns = t1 + t2
hdu = pyfits.BinTableHDU.from_columns(new_columns)
hdu.writeto(outtable)
# explicitly close the HDUs; might free up memory
hdulist1.close()
hdulist2.close()
  • Thank you Tom and Evert! Tom, that answer solves my problem. Evert, Earlier I did those steps but did not work. – vinu Nov 23 '15 at 13:28
  • That's a good explanation of the problem, thanks. This has revealed a subtle bug that I think I'll try to post a fix for. However, Tom Aldcroft's answer to use the higher level Astropy Table interface for table manipulation is the way to go for newer code. – Iguananaut Nov 24 '15 at 18:10
  • FWIW there's an issue open for this here: https://github.com/astropy/astropy/issues/4345#issuecomment-180957070 – Iguananaut Feb 08 '16 at 10:37