0

This is really confusing me:

Basically:

  1. I prepared a data file on HDFS;
  2. I create a new hive without location info;
  3. I load the data into the hive table; It works, the hive table is populated with right data;

I drop the hive table and start again:

  1. I did not upload HDFS file again as it is already there;
  2. I create exactly the same table with the same script;
  3. I attempted to load data into the same table I was rejected by this error:

FAILED: SemanticException Line 1:17 Invalid path '"data96/techie.txt"': No files matching path hdfs://nn01.itversity.com:8020/user/paslechoix/data96/techie.txt

For the entire output from hive, please see the file https://github.com/mdivk/175Scala/blob/master/reviews/first_ok_repeat_fail.txt.

John
  • 1,210
  • 5
  • 23
  • 51
Choix
  • 555
  • 1
  • 12
  • 28
  • Have you confirmed the file exists on given location? Also, try mentioning the complete hdfs path in your query. – shriyog Mar 24 '18 at 07:29
  • Thank you narush and Bala(below): the root cause is just as what you guys suspect: the data was "gone" right after it is loaded into the hive table - that is an interesting design – Choix Mar 24 '18 at 12:40

2 Answers2

0

When you drop a managed table, the data will also be deleted when the table is dropped. But on an external table only the table will be deleted.

Check this for more clarity

External vs Internal

Bala
  • 11,068
  • 19
  • 67
  • 120
  • Thank you and narush for the hint, the root cause is just as what you guys suspect: the data was "gone" right after it is loaded into the hive table - that is an interesting design – Choix Mar 24 '18 at 12:39
0

The reason it fails after the first successful load is because the file gets moved to the directory of the table you are creating. There would be no data on the original path after it gets loaded to the table. Check out HDFS path where you moved data to initially.

If you have to repeat the process, copy the file onto the HDFS path again and load the table.

Vamsi Prabhala
  • 48,685
  • 4
  • 36
  • 58