Drop rows with masked elements in astropy Table

Question

Consider an astropy Table() object.

from astropy.io import ascii

weather_data = """
  day,precip,type
  ,1.5,rain
  Tues, 9.,
  Wed,1.1,snow
  ed,,aaa
  Wd,1.1,snow
  """

dat = ascii.read(weather_data)
print(dat)

day  precip type
---- ------ ----
  --    1.5 rain
Tues    9.0   --
 Wed    1.1 snow
  ed     --  aaa
  Wd    1.1 snow

I need to drop all rows that contain at least one masked element. The final table should then look like:

day  precip type
---- ------ ----
 Wed    1.1 snow
  Wd    1.1 snow

This seems like a simple task, but I could not find a way t do it in the docs.

Add

I know that I could use .to_pandas() to convert the table to a pandas object and then use something like .dropna(), but this forces me to have pandas installed which I don't want.

rafaelc · Accepted Answer · 2018-05-09T19:54:53.590

3

For a generalized way

import operator
# if python3: from functools import reduce
dat[reduce(operator.and_, [~dat[col].mask for col in dat.columns])]

day   precip type
str4 float64 str4
---- ------- ----
 Wed     1.1 snow
  Wd     1.1 snow

What this process does is simply a generalized way of doing

dat[(~dat['day'].mask) & (~dat['precip'].mask) & (~dat['type'].mask)]

edited May 09 '18 at 19:54

answered May 09 '18 at 14:55

rafaelc

57,686
15
58
82

2

Thank you, this works perfectly. It shouldn't be necessary to do this though, I'll open an issue about this. – Gabriel May 09 '18 at 15:01

score 1 · Answer 2 · answered May 09 '18 at 18:33

1

Here is another way:

dat.remove_rows(np.where([c.data for c in dat.mask.itercols()])[-1])

But I agree with you that this filtering of invalid values should be supported by a masked table.

answered May 09 '18 at 18:33

AGN Gazer

8,025
2
27
45

This is also a great answer. The difference is that this one changes the table in place. – Gabriel May 09 '18 at 18:58

Drop rows with masked elements in astropy Table

2 Answers2