how Merging File from Local with File HDFS?

Question

i would like merging from local file from /opt/one.txt with file at my hdfs hdfs://localhost:54310/dummy/two.txt.

contains at one.txt : f,g,h
contains at two.txt : 2424244r

my code :

val cfg = new Configuration()
    cfg.addResource(new Path("/usr/local/hadoop/etc/hadoop/core-site.xml"))
    cfg.addResource(new Path("/usr/local/hadoop/etc/hadoop/hdfs-site.xml"))
    cfg.addResource(new Path("/usr/local/hadoop/etc/hadoop/mapred-site.xml"))

    try
    {
      val srcPath = "/opt/one.txt"
      val dstPath = "/dumCBF/two.txt"

      val srcFS = FileSystem.get(URI.create(srcPath), cfg)
      val dstFS = FileSystem.get(URI.create(dstPath), cfg)

      FileUtil.copyMerge(srcFS,
        new Path(srcPath),
        dstFS,
        new Path(dstPath),
        true,
        cfg,
        null)

      println("end proses")
    }

    catch
    {
      case m:Exception => m.printStackTrace()
      case k:Throwable => k.printStackTrace()
    }

i was following tutorial from : http://deploymentzone.com/2015/01/30/spark-and-merged-csv-files/

and it's not working at all, error would below this :

java.io.FileNotFoundException: File does not exist: /opt/one.txt

i dont why, sounds of error like that? BTW the file one.txt is exist

and then, im add some code to check exist file :

if(new File(srcPath).exists()) println("file is exist")

any idea or references? thanks!

EDIT 1,2 : typo extensions

Can you try `file:///opt/one.txt`, since its available in local file system. — Shankar, Nov 02 '16 at 07:08
hello mr. @Shankar yes, its worked. but now, back to topic about merging. it wouldn't merge at all, and without error anything :(. any idea why? — SunBright, Nov 02 '16 at 07:26
Before merging your files, you need to copy the local file to HDFS, then you can merge. — Shankar, Nov 02 '16 at 08:16
file two.txt i was upload to hdfs, should i upload too one.txt? — SunBright, Nov 02 '16 at 08:19
you dont have to copy the two.txt, because that's what you are going to create as output file, basically this command is used to merge multiple partitioned files to single file. see http://stackoverflow.com/questions/5700068/merge-output-files-after-reduce-phase and http://www.programcreek.com/java-api-examples/org.apache.hadoop.fs.FileUtil — Shankar, Nov 02 '16 at 08:23

how Merging File from Local with File HDFS?

0 Answers0