45

I tried to install hive on a raspberry pi 2. I installed Hive by uncompress zipped Hive package and configure $HADOOP_HOME and $HIVE_HOME manually under hduser user-group I created. When running hive, I got the following error message: hive

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql)

So I ran the command suggested in the above error message: schematool -dbType derby -initSchema I got the error message:

Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! * schemaTool failed *

It seems there aren't any helpful information when I try to google this error online. Any help or any explanation on how Hive works with Derby would be appreciated!

As high as honor
  • 451
  • 1
  • 5
  • 3

5 Answers5

145

After installing hive, if the first thing you did was run hive, hive attempted to create/initialize the metastore_db, but apparently might not get it right. On that initial run, maybe you saw your error:

Exception in thread "main" java.lang.RuntimeException: Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql)

Running hive, even though it fails, creates a metastore_db directory in the directory from which you ran hive:

ubuntu15-laptop: ~ $>ls -l |grep meta
drwxrwxr-x 5 testuser testuser 4096 Apr 14 12:44 metastore_db

So when you then tried running

ubuntu15-laptop: ~ $>schematool -initSchema -dbType derby

The metastore already existed, but not in complete form.

Soooooo the answer is:

  1. Before you run hive for the first time, run

    schematool -initSchema -dbType derby

  2. If you already ran hive and then tried to initSchema and it's failing:

    mv metastore_db metastore_db.tmp

  3. Re run

    schematool -initSchema -dbType derby

  4. Run hive again

**Also of note: if you change directories, the metastore_db created above won't be found! I'm sure there's a good reason for this that I don't know yet because I'm literally trying to use hive for the first time today. Ahhh here's information on this: metastore_db created wherever I run Hive

RJ Cole
  • 2,592
  • 3
  • 18
  • 23
  • 3
    Note that you need to restart derby after deleting or moving metastore_db folder before rerunning schema tool. Otherwise, you will get the same error on rerun. – skaarthik Jun 20 '16 at 21:50
  • 1
    Nice explanation. (Not just press that switch and magic will happen) – gnsb Nov 21 '16 at 08:00
  • 2
    O MY GOD, why is this not mentioned in official docs. Thank you! – Ayush Mar 05 '17 at 05:14
  • @Rebecca were you able to find out how to set the metastore permanently so that we don't create new metastore_db folder everytime invoking from a new directory the hive script. – Saurabh Mishra Mar 22 '17 at 14:32
  • For those looking for answer to fix the metastore_db location, look at this answer http://stackoverflow.com/a/13651963/5679728 – Saurabh Mishra Mar 22 '17 at 14:53
  • Note that this problem happens with other databases as well. If you are using mysql, do: 'drop database metastore; create database metastore' instead of removing a Derby file. – Craig S. Anderson Mar 27 '17 at 22:08
  • the "also of note" save my life: If I run `hive` in different directory, it sometimes works well(After I init schema), but another times, it does not work, because it will auto generate a `metastore_db` which is not properly configured. – carton.swing Mar 30 '20 at 18:03
  • At last I found the reason why inconsistency for run `hive` in different directory, I should use an absolute path: `/an/absolute/path/metastore_db` for derby instead of relative path: `metastore_db` – carton.swing Mar 30 '20 at 18:09
  • THANKYOU SIR Cole! – shawn1912 Oct 15 '21 at 16:35
19

I installed hive with HomeBrew(MacOS) at /usr/local/Cellar/hive and afer running schematool -dbType derby -initSchema I get the following error message:

Starting metastore schema initialization to 2.0.0 Initialization script hive-schema-2.0.0.derby.sql Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=30000) org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !!

However, I can't find either metastore_db or metastore_db.tmp folder under install path, so I tried:

  1. find /usr/ -name hive-schema-2.0.0.derby.sql
  2. vi /usr/local/Cellar/hive/2.0.1/libexec/scripts/metastore/upgrade/derby/hive-schema-2.0.0.derby.sql
  3. comment the 'NUCLEUS_ASCII' function and 'NUCLEUS_MATCHES' function
  4. rerun schematool -dbType derby -initSchema, then everything goes well!

hope this help someone.

dasu
  • 219
  • 2
  • 5
  • this is a great answer. How about using it to my question over here: http://stackoverflow.com/questions/43947930/unable-to-initialize-hive-with-derby-from-brew-install – WestCoastProjects May 13 '17 at 00:54
15

I encountered a similar issue, which I fixed by removing the derby database files rm -Rf $HIVE_HOME/metastore_db

Be aware, this will remove your schema completely!

After cleaning the old schema I could reinitialize with:

$HIVE_HOME/bin/schematool -initSchema -dbType derby

It might differ from where you call hive, try to go to your hive installation directory and run ./bin/hive

michiel
  • 6,036
  • 1
  • 18
  • 13
2

Try this one: schematool -dbType mysql -initSchema

Read the documentation of Hive SchemTool: https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

Hamdi Charef
  • 639
  • 2
  • 12
  • 19
0

If you use Derby, you can remove DB files and run the init process.

Or if problem persists, you can edit the SQL files Hive use. For example:

/usr/hdp/2.5.6.0-40/hive2/scripts/metastore/upgrade/mysql/ contains those for mysql.

Thomas Decaux
  • 21,738
  • 2
  • 113
  • 124