4

I am reading in a csv into spark using SpraklyR

schema <- structType(structField("TransTime", "array<timestamp>", TRUE),
                 structField("TransDay", "Date", TRUE))

 spark_read_csv(sc, filename, "path", infer_schema = FALSE, schema = schema)

But get: 

Error: could not find function "structType"

How do I specify colunm types using spark_read_csv?

Thanks in advance.

Levi Brackman
  • 325
  • 2
  • 17

2 Answers2

7

The structType function is from Scala's SparkAPI, in Sparklyr to specify the datatype you must pass it in the "column" argument as a list, suppose that we have the following CSV(data.csv):

name,birthdate,age,height
jader,1994-10-31,22,1.79
maria,1900-03-12,117,1.32

The function to read the corresponding data is:

mycsv <- spark_read_csv(sc, "mydate", 
                          path =  "data.csv", 
                          memory = TRUE,
                          infer_schema = FALSE, #attention to this
                          columns = list(
                            name = "character",
                            birthdate = "date", #or character because needs date functions
                            age = "integer",
                            height = "double"))
# integer = "INTEGER"
# double = "REAL"
# character = "STRING"
# logical = "INTEGER"
# list = "BLOB"
# date = character = "STRING" # not sure

For manipulating datetype you must use the hive date functions, not R functions.

mycsv %>% mutate(birthyear = year(birthdate))

Reference: https://spark.rstudio.com/articles/guides-dplyr.html#hive-functions

Jader Martins
  • 759
  • 6
  • 26
  • 1
    Any ideas about bigint / int64 /long? – nachti Dec 18 '17 at 16:01
  • @nachti the translation is done here https://github.com/rstudio/sparklyr/blob/6d49608d7ce54e89a201677ec04d2083b38ab375/R/dbi_spark_connection.R#L36 as you can see no long type. – Jader Martins Feb 18 '18 at 14:48
2

we have an example of how to do that in one of our articles in the official sparklyr site, here is the link: http://spark.rstudio.com/example-s3.html#data_import

edgararuiz
  • 625
  • 5
  • 9