0

I am trying to checkpoint my spark streaming context to hdfs to handle a failure at some point of my application. I have my HDFS setup on a separate cluster and spark running on a separate standalone server. To do this, I am using : 

ssc.checkpoint(directory: String)

This gives me : org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE when I try with directory as "hdfs://hostname:port/pathToFolder"

How can I checkpoint to a remote hdfs path? Is it possible to add credentials to the string uri? I tried googling, but no help so far.

Thanks and appreciate any help!

void
  • 2,403
  • 6
  • 28
  • 53

1 Answers1

0

You can provide the credentials by using:

hdfs://username:password@hostname:port/pathToFolder
Marco
  • 4,837
  • 2
  • 22
  • 27