pyspark counting the numbers of NA in each column and group by context

Asked Jul 29 '18 at 19:03

Active Jul 29 '18 at 23:54

Viewed 469 times

I am working on databricks project and trying to count for each column how many nulls there are and group it by specific context (sub_area)

nulls can be in a form of : None, NaN and NA. i know there are functions such as isnan under

from pyspark.sql.functions import col, count, isnan,

how do i count the number of NA value in each column, or a given specific value?

edited Jul 29 '18 at 23:54

zero323

asked Jul 29 '18 at 19:03

Ohad Gerrassy

Possible duplicate of [SparkSQL: apply aggregate functions to a list of column](https://stackoverflow.com/questions/33882894/sparksql-apply-aggregate-functions-to-a-list-of-column) – zero323 Jul 29 '18 at 19:37
how do i count a specific value in each column? – Ohad Gerrassy Jul 29 '18 at 20:07
[Count number of non-NaN entries in each column of Spark dataframe with Pyspark](https://stackoverflow.com/q/33900726) – zero323 Jul 29 '18 at 23:54
This is for non nulls.... i need a specific value – Ohad Gerrassy Jul 30 '18 at 11:14

0 Answers0