How to access N Chunk using Pandas?

Question

i'm sure most of you might find this basic but i'm somehow finding it very confusing to understand the way to access a particular chunk in pandas and append it later. I know to append the set but i don't know to identify the data based on a chunk

for ex, just imagine my table has 36000 records and i chunk it by 1200, now i want to access just the 3rd chunk only. how to achieve it in pandas? i googled it extensively but no good results

for df in pd.read_sql_query('select id from table;', conn, chunksize=1200):
          print(df)

Does this answer your question? https://stackoverflow.com/questions/44729727/pandas-slice-large-dataframe-in-chunks — Ehsan, Aug 22 '20 at 08:11
you mean limit the size of the records at sql level before ingesting into pandas? — , Aug 22 '20 at 08:11

score 0 · Answer 1 · answered Aug 22 '20 at 08:25

Pandas - Slice Large Dataframe in Chunks

thank you for pointing out to this link. The fix is pretty simple!

df = pd.read_sql_query('select * from x',conn)

 chunksize=100

new_df= [df[i: i+n] for i in range(0,df.shape[0],chunksize)]

now if you add the index with your frame and print it, you can see your chunk

  new_df[0] ----prints the data in the 1st chunk
  new_df[1] ----prints the data in 2nd chunk
  new_df[2] ----prints the data in the 3rd chunk

How to access N Chunk using Pandas?

1 Answers1