0

We are experiencing a problem while converting our R dataframe to Spark data frame. We employ 'collect()' function to load our data to R from Spark. Later after finishing data manipulation we transfer our table to Spark by using copy_to() function. But, In our output, date column appears in dbl format. Before skipping to Spark in R, it was in date format. How can we maintain our date variables format as date, while converting to Spark data frame.

df_spark <- copy_to(spark_connect, df, overwrite = TRUE)
filter(df_spark, ID_number == 1234)
        ID_number        date
        <int>            <dbl>        
         1234           17307
omzeybek
  • 305
  • 2
  • 14
  • We tried the solution referred in that posting. Bur to_date() function in SparkR does not work for us – omzeybek Mar 20 '19 at 11:41
  • There is not a single piece of SparkR code in the linked question. `to_date` is a component of the [SQL translation layer](https://dbplyr.tidyverse.org/articles/sql-translation.html). And in general [SparkR and sparklyr are not compatible and shouldn't be used in the same project](https://stackoverflow.com/q/40577650/10938362) . – user10938362 Mar 20 '19 at 11:43
  • That's not SparkR code. Please refer to the linked documentation, to understand how this (and `sparklyr` in general) works. You can also check [this](https://stackoverflow.com/q/52972504/10938362) for more `sparklyr` focused explanation. – user10938362 Mar 20 '19 at 11:45

0 Answers0