0

I want to import data from Mysql using sqoop import but my requirement is i want to use 4 mappers but it should create only one file in hdfs target directory is there is any way to do this ?

Rajendra B
  • 11
  • 5
  • Possible duplicate of [merge output files after reduce phase](http://stackoverflow.com/questions/5700068/merge-output-files-after-reduce-phase) – Rahul Sharma Oct 25 '16 at 17:07

3 Answers3

2

No. there is no option in sqoop to re-partition files into 1 file.

I don't think this should be a headache of sqoop.

You can do it easily using getmerge feature of hadoop. Example:

hadoop fs -getmerge /sqoop/target-dir/ /desired/local/output/file.txt

Here

  • /sqoop/target-dir is the target-dir of your sqoop command (directory containing all the part files).

  • desired/local/output/file.txt is the combined single file.

Dev
  • 13,492
  • 19
  • 81
  • 174
0

you can use below sqoop command..!!

Suppose database name is prateekDB and table name is Emp...!!

sqoop import  --connect "jdbc:mysql://localhost:3306/prateekDB"  --username=root \
--password=data  --table Emp  --target-dir /SqoopImport  --split-by empno
Piyush Patel
  • 1,646
  • 1
  • 14
  • 26
Prateek Jaiswal
  • 1
-1

Add this option to sqoop 

--num-mappers 1

the sqoop log shows:

Job Counters 
    Launched map tasks=1
    Other local map tasks=1

and finally on hdfs ONE file is created.

ozw1z5rd
  • 3,034
  • 3
  • 32
  • 49