Apache Kylin Unable to find HBase common lib

Question

I have installed Hadoop version 2.6.0, HBase version 0.99.0 , Hive version 1.2, Kylin version 1.5.0.

I have setup all of the above in Standalone mode while in running Kylin it checks in early stage about Hadoop, HBase and Hive. Each and everything has been installed but when I start Kylin it give an error of HBase common lib not found. Following is the log of Apache Kylin.

KYLIN_HOME is set to bin/../
16/03/24 18:02:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
kylin.security.profile is set to testing
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Downloads/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/kunalgupta/Downloads/spark-1.6.0-bin-hadoop2.6/lib/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
HIVE_CONF is set to: /Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/conf/, use it to locate hive configurations.
HCAT_HOME is set to: /Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/hcatalog, use it to find hcatalog path:
usage: dirname path
find: -printf: unknown primary or operator
hive dependency: /Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/conf/::/Users/kunalgupta/Desktop/kunal/Desktop/Backup/Kunal/Downloads/apache-hive-1.2.1-bin/hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1.jar
cut: illegal option -- -
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-d delim] [file ...]
hbase-common lib not found

Please somebody help me out.

score 2 · Accepted Answer · edited Apr 13 '17 at 12:45

The issue you encountered is that cut command on mac-osx does not support "--output-delimiter" option. I encountered the same error while installing kylin-1.5.1. The easy to solve it is to use the gnu binaries in your shell instead of osx binaries.

Use brew for installing coreutils (I changed all the commonly used shell utils to their gnu versions)
Use the below command for that.

brew install coreutils findutils gnu-tar gnu-sed gawk gnutls gnu-indent gnu-getopt --default-names

Now to make the shell use these instead of mac binaries add the path to these utils in your shell profile file.

vi ~/.profile

add the following lines to this file

PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"

After this open a new terminal window and do

echo $PATH

The result should have the path that we set in the previous step in the front (prepended) Now start kylin, should work smoothly.

Some references links that helped me:
Mac forum link
Installation guide from apple.se

score 1 · Answer 2 · answered May 10 '19 at 09:46

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export KYLIN_HOME=/usr/local/kylin
export CDH_HOME=/opt/cloudera/parcels/CDH
export SPARK_HOME=${CDH_HOME}/lib/spark
export HBASE_HOME=${CDH_HOME}/lib/hbase
export HIVE_HOME=${CDH_HOME}/lib/hive
export HADOOP_HOME=${CDH_HOME}/lib/hadoop
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

source /etc/profile

added ${HBASE_HOME} to /etc/profile

cat /opt/cloudera/parcels/CDH/lib/hbase/bin/hbase

if [ -n "${shaded_jar}" ] && [ -f "${shaded_jar}" ]; then
  CLASSPATH="${CLASSPATH}:${shaded_jar}"
# fall through to grabbing all the lib jars and hope we're in the omnibus tarball
#
# N.B. shell specifically can't rely on the shaded artifacts because RSGroups is only
# available as non-shaded
#
# N.B. pe and ltt can't easily rely on shaded artifacts because they live in hbase-mapreduce:test-jar
# and need some other jars that haven't been relocated. Currently enumerating that list
# is too hard to be worth it.
#
else
  for f in $HBASE_HOME/lib/*.jar; do
    CLASSPATH=${CLASSPATH}:$f;
  done
  # make it easier to check for shaded/not later on.
  shaded_jar=""
fi
for f in "${HBASE_HOME}"/lib/client-facing-thirdparty/*.jar; do
  if [[ ! "${f}" =~ ^.*/htrace-core-3.*\.jar$ ]] && \
     [ "${f}" != "htrace-core.jar$" ] && \
     [[ ! "${f}" =~ ^.*/slf4j-log4j.*$ ]]; then
    CLASSPATH="${CLASSPATH}:${f}"
  fi
done

score 1 · Answer 3 · answered Sep 30 '19 at 02:05

I use centos7 ,apache-kylin-3.0.0-alpha2-bin-hadoop3 and hbase 2.1.4 . I found solution in these links. https://issues.apache.org/jira/browse/KYLIN-3564 http://92072234.wiz03.com/share/s/2i1O8Q1L1k042IDoOy3h7BgH2K4G6J2SoQv42Xc4b01xpCrj The second link solve my problem. Edit hbase file

$HBASE_HOME/bin/hbase

change

CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar

to

CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar:/YOUR HBASE FULL PATH or $HBASE_HOME/lib/*

score 0 · Answer 4 · answered Jan 08 '20 at 09:58

0

I solved it as the following:

1. export HBASE_CLASSPATH=/opt/cloudera/parcels/CDH-6.2.0-/hbase-common-2.0.0-cdh6.2.0.jar
2. then start kylin again.

answered Jan 08 '20 at 09:58

DennisLi

3,915
6
30
66

score -1 · Answer 5 · answered Mar 26 '16 at 13:11

-1

You are running on Windows? Sorry Kylin only runs on Linux as of version 1.5

answered Mar 26 '16 at 13:11

Li Yang

284
1
6

Thanks @Li Yang ,I am running it on Mac OSX... so should I downgrade the version of Kylin to run on Mac osX EI Capitan – Kunal Gupta Mar 27 '16 at 05:15
Kylin was not tested on Mac / Windows. For testing or dev, I recommend run Kylin in a Hadoop sandbox. – Li Yang Mar 30 '16 at 10:10
Btw the root cause of your problem is the 'cut' command on Mac does not support "--output-delimiter" option which is used in a Kylin shell script. This bug might be fixed, but running Kylin on Mac is still a bad idea because no one does that. – Li Yang Mar 30 '16 at 10:14

Apache Kylin Unable to find HBase common lib

5 Answers5