Spark SQL: IllegalArgumentException: Wrong FS, expected: file:///

Question

I try to put data in Hive table via Spark SQL terminal.

I created table based on Avro Schema .avsc :

CREATE TABLE test_table
  ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
  STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
  OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
  TBLPROPERTIES (
    'avro.schema.url'='file:///C:/some_dir/table_schema.avsc');

Then I try to load data from Avro file which stored in HDFS:

LOAD DATA INPATH 'hdfs://localhost:9000/data/test_table.avro' OVERWRITE INTO TABLE test_table;

I get an error:

Error in query: java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9000/data/test_table.avro, expected: file:///;

When I try to load from local file system it works.

Also my core-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
        <final>true</final>
        <description>NameNode URI</description>
    </property>
</configuration>

How to load data from files which stored in HDFS?

try `LOAD DATA INPATH 'hdfs:///localhost:9000/data/test_table.avro' OVERWRITE INTO TABLE test_table;` — Ramesh Maharjan, Jun 28 '17 at 12:24
@ramesh-maharjan ava.io.IOException: Incomplete HDFS URI, no host: hdfs:/localhost:9000/data/test_table.avro — Bo., Jun 28 '17 at 13:14
@ramesh-maharjan fs.defaultFS hdfs://localhost:9000 true NameNode URI — Bo., Jun 28 '17 at 14:04
why don''t you try with scala ? more info https://www.cloudera.com/documentation/enterprise/5-7-x/topics/spark_avro.html — Ramesh Maharjan, Jun 28 '17 at 15:12
I just would like to figure out this way. I know that way works.. probably forgot some small detail in the settings.. — Bo., Jun 28 '17 at 16:43

Spark SQL: IllegalArgumentException: Wrong FS, expected: file:///

0 Answers0