9

I am new to Hadoop and I just started working on Hive, I my understanding it provides a query language to process data in HDFS. With HiveQl we can create tables and load data into it from HDFS.

So my question is: where are those tables stored? Specifically if we have 100 GB file in our HDFS and we want to make a hive table out of that data what will be the size of that table and where is it stored?

If my understanding about this concept is wrong please correct me ..

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
talin
  • 179
  • 3
  • 12

2 Answers2

5

If the table is 100GB you should consider an Hive External Table (as opposed to a "managed table", for the difference, see this).

With an external table the data itself will be still stored on the HDFS in the file path that you specify (note that you may specify a directory of files as long as they all have the same structure), but Hive will create a map of it in the meta-store whereas the managed table will store the data "in Hive".

When you drop a managed table, it drops the underlying data as opposed to dropping a hive external table which only drops the meta-data from the meta-store referencing that data.

Either way you are using only 100GB as viewed by the user and are taking advantage of the HDFS' robustness though duplication of the data.

Community
  • 1
  • 1
mlegge
  • 6,763
  • 3
  • 40
  • 67
  • 1
    What do you mean "load the entire table into memory and map it in its meta-store"? Hive doesn't do anything like that when you create a managed table. – Jeremy Beard Mar 27 '15 at 01:18
  • Then, what is really happing there while we load data into a hive table ? – talin Mar 27 '15 at 05:59
  • @JeremyBeard If my semantics are wrong, please feel free to edit the post. If the entire concept is wrong I would very much like to know where my understanding isn't correct so I would encourage you to submit an answer – mlegge Mar 27 '15 at 12:19
  • You can add files to the managed table directory as well as external. Have you ever tried this? It works good. You are wrong in the very first sentence about 'loading in memory' – leftjoin Dec 25 '16 at 09:08
1

Hive will create a directory on HDFS. If you didn't specify any location it will create a directory at /user/hive/warehouse on HDFS. After load command the files are moved to the /warehouse/tablename. You can also point to the HDFS directory if it contains partitions (if the files are partitioned), or use external table concept.

Tim
  • 12,318
  • 7
  • 50
  • 72
Sravan K Reddy
  • 1,082
  • 1
  • 10
  • 19