Sqoop + S3 + Parquet results in Wrong FS error

Question

When trying to import data to S3 in Parquet format using Sqoop, as follows:

bin/sqoop import --connect 'jdbc:[conn_string]' --table [table] --target-dir s3a://bucket-name/ --hive-drop-import-delims --as-parquetfile

... I get the following error:

ERROR tool.ImportTool: Imported Failed: Wrong FS: s3a://bucket-name/, expected: hdfs://localhost:9000

I have no problem importing non-parquet data or working with s3a directly through HDFS. Seems like this issue, but it was supposedly fixed many versions ago.

Which version of Sqoop are you using? – Suraj Nayak Oct 23 '15 at 01:41 — Suraj Nayak, Oct 23 '15 at 01:41
Is there a ticket against sqoop for this? – spazm Feb 28 '18 at 20:51 — spazm, Feb 28 '18 at 20:51

score 3 · Answer 1 · answered Mar 14 '17 at 11:09

3

Sqoop import in parquet format currently doesn't support s3 as target directory, work around you can use is to import data to hdfs and then use distcp to copy the from hdfs to s3

answered Mar 14 '17 at 11:09

amar vakacharla

61
4

score 0 · Answer 2 · answered Apr 20 '16 at 23:48

0

Sqoop doesn't support import to parquet file if you are using S3 as a target directory.

answered Apr 20 '16 at 23:48

dreamer

1,039
2
16
36

Sqoop + S3 + Parquet results in Wrong FS error

2 Answers2

Linked