1

I have asked the very same question here and got a nice answer for dplyr but now I want to do the same on sparklyr.

So I have a spark dataframe where one of the columns contains vectors/lists. I'd like to create a column that contain the element of the vector with index i (an exterior variable).

e.g. with i = 2

link to data

In dplyr the solution is:

data <- data %>%
  mutate(element = sapply(list_x, "[", i))

But this doesn't work in sparklyr as sapply doesn't exist in sparklyr.

vl237611
  • 51
  • 3
  • Without providing the structure, it is difficult to comment. Did you convert to spark dataset, like `iris_tbl <- copy_to(sc, iris)`; What is `data`? – akrun Jul 02 '18 at 18:01
  • Yes, data is a spark dataframe. Sorry for the confusion – vl237611 Jul 02 '18 at 18:23
  • Without knowing how data is created (what is its Spark type) it is hard to answer. While dput might not be a good option for Spark dataframes you should provide some form of reproducible example. – Alper t. Turker Jul 04 '18 at 16:03

0 Answers0