This is an easy question but say I have an MxN matrix. All I want to do is extract specific columns and store them in another numpy array but I get invalid syntax errors. Here is the code:
extractedData = data[[:,1],[:,9]].
It seems like the above line should suffice but I guess not. I looked around but couldn't find anything syntax wise regarding this specific scenario.
I assume you wanted columns 1 and 9?
To select multiple columns at once, use
X = data[:, [1, 9]]
To select one at a time, use
x, y = data[:, 1], data[:, 9]
With names:
data[:, ['Column Name1','Column Name2']]
You can get the names from data.dtype.names…
Assuming you want to get columns 1 and 9 with that code snippet, it should be:
extractedData = data[:,[1,9]]
if you want to extract only some columns:
idx_IN_columns = [1, 9]
extractedData = data[:,idx_IN_columns]
if you want to exclude specific columns:
idx_OUT_columns = [1, 9]
idx_IN_columns = [i for i in xrange(np.shape(data)[1]) if i not in idx_OUT_columns]
extractedData = data[:,idx_IN_columns]
Just:
>>> m = np.matrix(np.random.random((5, 5)))
>>> m
matrix([[0.91074101, 0.65999332, 0.69774588, 0.007355 , 0.33025395],
[0.11078742, 0.67463754, 0.43158254, 0.95367876, 0.85926405],
[0.98665185, 0.86431513, 0.12153138, 0.73006437, 0.13404811],
[0.24602225, 0.66139215, 0.08400288, 0.56769924, 0.47974697],
[0.25345299, 0.76385882, 0.11002419, 0.2509888 , 0.06312359]])
>>> m[:,[1, 2]]
matrix([[0.65999332, 0.69774588],
[0.67463754, 0.43158254],
[0.86431513, 0.12153138],
[0.66139215, 0.08400288],
[0.76385882, 0.11002419]])
The columns need not to be in order:
>>> m[:,[2, 1, 3]]
matrix([[0.69774588, 0.65999332, 0.007355 ],
[0.43158254, 0.67463754, 0.95367876],
[0.12153138, 0.86431513, 0.73006437],
[0.08400288, 0.66139215, 0.56769924],
[0.11002419, 0.76385882, 0.2509888 ]])
One thing I would like to point out is, if the number of columns you want to extract is 1 the resulting matrix would not be a Mx1 Matrix as you might expect but instead an array containing the elements of the column you extracted.
To convert it to Matrix the reshape(M,1) method should be used on the resulting array.
One more thing you should pay attention to when selecting columns from N-D array using a list like this:
data[:,:,[1,9]]
If you are removing a dimension (by selecting only one row, for example), the resulting array will be (for some reason) permuted. So:
print data.shape # gives [10,20,30]
selection = data[1,:,[1,9]]
print selection.shape # gives [2,20] instead of [20,2]!!
You can use the following:
extracted_data = data.ix[:,['Column1','Column2']]
Here is yet another example that some may find useful when you need specific columns and ranges from your data, this takes a few seconds to run on millions of rows and you can just add more columns by adding additional lists (e.g., columns = ... + [1] + [5], etc.:
columns = [0] + [x for x in range(4,62-3)]
print(columns)
selected_data = train_data[:,columns]
I think the solution here is not working with an update of the python version anymore, one way to do it with a new python function for it is:
extracted_data = data[['Column Name1','Column Name2']].to_numpy()
which gives you the desired outcome.
The documentation you can find here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.html#pandas.DataFrame.to_numpy
I could not edit the chosen answer so I'm adding an answer to clarify that using an integer to index seems to be returning a view (not a copy) while using a list returns a copy
>>> x = np.zeros(shape=[2, 3])
>>> y = x[:, [0, 1]]
>>> z1, z2 = x[:, 0], x[:, 1]
>>> y[0, 0] = 1
>>> print(y)
[[1. 0.]
[0. 0.]]
>>> print(x)
[[0. 0. 0.]
[0. 0. 0.]]
>>> z1[0] = 2
>>> print(z1)
[2. 0.]
>>> print(x)
[[2. 0. 0.]
[0. 0. 0.]]
