1

I do not know why I cannot move a file from one directory to another. I can view the content of the file but I cannot move the same file into another directory.

WORKS FINE:

hadoop fs -cat /user/hadoopusr/project-data.txt

DOES NOT WORK:

hadoop fs -put /user/hadoopusr/project-data.txt /user/hadoopusr/Projects/MarketAnalysis

I got a No such file or directory error message. What is wrong? Please help. Thank you!

Coursal
  • 1,387
  • 4
  • 17
  • 32
Francisco
  • 33
  • 5
  • I am trying to create a data frame in scala and when I refer to the file from scala I got this message: org.apache.spark.sql.AnalysisException: Path does not exist: file:/user/hadoopusr/project-data.txt; – Francisco Oct 29 '20 at 18:26
  • Can you show us the error message you get when you try the second command? And also the Hadoop (and/or Spark) release you are operating on? – Coursal Oct 29 '20 at 20:07
  • This is the error message I got in Spark: org.apache.spark.sql.AnalysisException: Path does not exist: file:/home/hadoopusr/Desktop/project-data.txt; – Francisco Oct 31 '20 at 21:31

1 Answers1

2

As we can read from here about the -put command:

This command is used to copy files from the local file system to the HDFS filesystem. This command is similar to –copyFromLocal command. This command will not work if the file already exists unless the –f flag is given to the command. This overwrites the destination if the file already exists before the copy

Which makes it clear why it doesn't work and throws the No such file or directory message. It's because it can't find any file with the name project-data.txt on your current directory of your local filesystem.

You plan on moving a file between directories inside the HDFS, so instead of using the -put parameter for moving, we can simply use the -mv parameter as we would in our local filesystem!

Tested it out on my own HDFS as follows:

  1. Create the source and destination directories in HDFS

hadoop fs -mkdir source_dir dest_dir

  1. Create an empty (for the sake of the test) file under the source directory

hadoop fs -touch source_dir/test.txt

  1. Move the empty file to the destination directory

hadoop fs -mv source_dir/test.txt dest_dir/test.txt

(Notice how the /user/username/part of the path for the file and the destination directory is not needed, because HDFS is by default on this directory where you are working. You also should note that you have to write the full path of the destination with name of the file included.)

You can see below with the HDFS browser that the empty text file has been moved to the destination directory: enter image description here

Coursal
  • 1,387
  • 4
  • 17
  • 32
  • Thanks for taking the time to teach me about Hadoop. It worked! Although, I still need to make it work in Spark . This is the error I got in Spark: org.apache.spark.sql.AnalysisException: Path does not exist: file:/home/hadoopusr/Desktop/project-data.txt; – Francisco Oct 31 '20 at 21:30
  • Glad to help. Feel free to write a new question about this particular issue, because it is not relevant enough to be answered here based on your question. Edit: found the following answer for your problem on Spark, I think you could benefit from it https://stackoverflow.com/a/53558836/5644037 – Coursal Oct 31 '20 at 22:07