How would a convenience function look like to convert a pandas dataframe to a structured numpy array?

Question

For the conversion of a pandas dataframe to a plain numpy array, I normally use the following convenience function:

def df2numpy(df):
    df.index.name = "i"
    valDf = df.values
    indDf = df.index
    colsDf = df.columns
    colDicDf = {}
    for runner in range(len(df.columns)):
        colDicDf[df.columns[runner]] = runner
    return valDf, indDf, colDicDf

This delivers to me the

• numpy array valDf,
• an array with the index indDf and
• a dict colDicDf that can be accessed easily via colDicDf["column_name"] to get the index of the column I am interested in.

How would the same look like - in general terms - if I would like to convert a dataframe to a structured array?

Some helpful input might be the following code (see When to use a numpy struct or a numpy record array? ):

import numpy as np
a = np.array([['2018-04-01T15:30:00'],
       ['2018-04-01T15:31:00'],
       ['2018-04-01T15:32:00'],
       ['2018-04-01T15:33:00'],
       ['2018-04-01T15:34:00']], dtype='datetime64[s]')
c = np.array([0,1,2,3,4]).reshape(-1,1)

# create the compound dtype
dtype = np.dtype(dict(names=['date', 'val'], formats=[arr.dtype for arr in (a, c)]))

# create an empty structured array
struct = np.empty(a.shape[0], dtype=dtype)

# populate the structured array with the data from your column arrays
struct['date'], struct['val'] = a.T, c.T

print(struct)
# output:
#     array([('2018-04-01T15:30:00', 0), ('2018-04-01T15:31:00', 1),
#            ('2018-04-01T15:32:00', 2), ('2018-04-01T15:33:00', 3),
#            ('2018-04-01T15:34:00', 4)],
#           dtype=[('date', '<M8[s]'), ('val', '<i8')])

Answer 1

Convert DataFrame to ndarray

Here's a general function for converting from a DataFrame to a structured ndarray:

import numpy as np
import pandas as pd

def frameToStruct(df):
    # convert dataframe to record array, then cast to structured array
    struct = df.to_records(index=False).view(type=np.ndarray, dtype=list(df.dtypes.items()))

    # return the struct and the row labels
    return struct, df.index.values

# example dataframe
df = pd.DataFrame(data=[[True, 1,2],[False, 10,20]], columns=['a','b','c'])

struct,rowlab = frameToStruct(df)

print(struct)
# output
#     [( True,  1,  2) (False, 10, 20)]

print(rowlab)
# output
#     [0 1]

# you don't need to keep track of columns separately, struct will do that for you
print(struct.dtype.names)
# output
#     ('a', 'b', 'c')

Why you should prefer structured arrays over record arrays

One good reason to use structure arrays instead of record arrays is that column access is much faster for a structured array:

# access record array column by attribute
%%timeit
rec.c
# 4.64 µs ± 79.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# get record array column
%%timeit
rec['c']
# 3.66 µs ± 29.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

# get structured array column
%%timeit
struct['c']
# 163 ns ± 4.39 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

See this book for more info.