Is there a Pandas equivalent to each_slice to operate on dataframes

Question

I am wondering if there is a Python or Pandas function that approximates the Ruby #each_slice method. In this example, the Ruby #each_slice method will take the array or hash and break it into groups of 100.

var.each_slice(100) do |batch|
  # do some work on each batch

I am trying to do this same operation on a Pandas dataframe. Is there a Pythonic way to accomplish the same thing?

I have checked out this answer: Python equivalent of Ruby's each_slice(count)

However, it is old and is not Pandas specific. I am checking it out but am wondering if there is a more direct method.

You could use numpy's `array_split` e.g. `np.array_split(df,100)` does this solve your problem? — EdChum, Aug 06 '14 at 20:10
Hmm. I think you'd need to do `np.array_split(df,df.shape[0]/100)` to split it into batches of ~100 in size — EdChum, Aug 06 '14 at 20:15
@EdChum that works great! You should submit it as an answer. — analyticsPierce, Aug 06 '14 at 20:36

score 1 · Accepted Answer · answered Aug 06 '14 at 20:40

1

There isn't a built in method as such but you can use numpy's array_slice, you can pass the dataframe to this and the number of slices.

In order to get ~100 size slices you'll have to calculate this which is simply the number of rows/100:

import numpy as np
# df.shape returns the dimensions in a tuple, the first dimension is the number of rows
np.array_slice(df, df.shape[0]/100)

This returns a list of dataframes sliced as evenly as possible

answered Aug 06 '14 at 20:40

EdChum

376,765
198
813
562

how do I perform logic on each of the slices? – analyticsPierce Aug 06 '14 at 20:44
I got it: for x in range(len(slices)) – analyticsPierce Aug 06 '14 at 20:47
You can just assign the output to a list and iterate over it like `slices = np.array_slice(df, df.shape[0]/100)` and then `for s in slices: dostuff` – EdChum Aug 06 '14 at 20:48

Is there a Pandas equivalent to each_slice to operate on dataframes

1 Answers1