Pyspark summarising the results

Asked Sep 21 '21 at 13:35

Active Sep 21 '21 at 13:39

Viewed 41 times

i have created a spark DF which include around 20 columns. All these are updated with 0, 1 . I need to aggregate the result.

For example:- I have col1 col2 …. All these are updated with 0,1 .

I want to pull out a summary as :-

Values| Col1 | col2 
  0     Count  count
  1     Count  count

That will give us, how many zeros are there in col1 and how many 1s.

edited Sep 21 '21 at 13:39

OneCricketeer

asked Sep 21 '21 at 13:35

puneet madan

Please share your code and tell us what difficulty you've had so far creating the output you expect – OneCricketeer Sep 21 '21 at 13:40
Just tried doing that on one column. Like sparkdf.groupby(“col1”).agg({“col” : ‘count}). However, need to run the steps on all columns and need summary in 2rows with 20columns with counts. – puneet madan Sep 21 '21 at 13:46
You could write a for loop over each of the column names – OneCricketeer Sep 21 '21 at 13:49
Please can you advise on this. I tried . For col in sparkdf.schema.names: – puneet madan Sep 21 '21 at 13:49
Does this example help? https://stackoverflow.com/a/56257865/2308683 – OneCricketeer Sep 21 '21 at 13:53
Yes. It does. Will give it a try on my dataset. Seems straight forward. Thanks – puneet madan Sep 21 '21 at 13:56

0 Answers0