0

I am trying to copy files from HDFS to S3 using distcp by executing the following command

hadoop distcp -fs.s3a.access.key=AccessKey -fs.s3a.secret.key=SecrerKey \
s3n://testbdr/test2 hdfs://hostname:portnumber/tmp/test

But I am getting the following error:

17/09/05 02:59:30 ERROR tools.DistCp: Invalid arguments:
java.lang.IllegalArgumentException: Both source file listing and source paths present
        at org.apache.hadoop.tools.OptionsParser.parseSourceAndTargetPaths(OptionsParser.java:341)
        at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:89)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:112)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:436)
Invalid arguments: Both source file listing and source paths present
usage: distcp OPTIONS [source_path...] <target_path>
              OPTIONS
 -append                       Reuse existing data in target files and
                               append new data to them if possible
 -async                        Should distcp execution be blocking
philantrovert
  • 9,904
  • 3
  • 37
  • 61
Vinod Mhetre
  • 3
  • 3

2 Answers2

0

To pass configuration parameters, you have to use -D

hadoop distcp -Dfs.s3a.access.key=AccessKey -Dfs.s3a.secret.key=SecrerKey \
s3n://testbdr/test2 hdfs://hostname:portnumber/tmp/test
philantrovert
  • 9,904
  • 3
  • 37
  • 61
  • tried above command . now i am getting below error.. java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively). – Vinod Mhetre Sep 06 '17 at 09:53
  • https://stackoverflow.com/questions/24924808/how-to-specify-aws-access-key-id-and-secret-access-key-as-part-of-a-amazon-s3n-u – philantrovert Sep 06 '17 at 09:55
0

Old Command

hadoop distcp -Dfs.s3a.access.key=AccessKey -Dfs.s3a.secret.key=SecretKey \ s3n://testbdr/test2 hdfs://hostname:portnumber/tmp/test

Rectified Command

hadoop distcp -Dfs.s3n.awsAccessKeyId=AccessKey -Dfs.s3n.awsSecretAccessKey=SecretKey \ s3n://testbdr/test2 hdfs://hostname:portnumber/tmp/test

Vinod Mhetre
  • 3
  • 3