I- Summary:
I run into an issue while testing Hadoop (2.6.0)/Hbase (1.0.1.1) on a fully distributed mode. The problem is that everytime I create a Table, the query freezes, with the error "handler.CreateTableHandler : failed. null". Even though, when I check, the table is successfully created.
II- Set-up and configuration :
I have a quit simple setup of 5 machines; one namenode/HMaster, one resourceManager/ZKQuorum, and three slaves, two of them are region servers, while the third is a backup HMaster.
III- Porblem
Hadoop is up and running, but HBase surprisingly freezes after I create a table. The behaviour remains the same either by doing it through HBase shell, or Java API. Despite the freezing part, the table is everytime created.
As you can notice at the last log entry below, the createTableHandler fails with the msg null.
I looked around for posts similar to mine, I only found 2. Whereas the first one has no answer, in the second, a suggestion was to reboot the server. Indeed, I tried this solution, but unfortunately it didn't work for me. Other solutions I tried were:
- Cleaning ZK
- Hbase offlineMetaRepair
- Assigning HMaster to different machines.
Does anybody have an idea where this could come from ?
Thank you so much in advance!
2015-07-08 20:43:36,645 INFO[B.defaultRpcServer.handler=3,queue=0,port=16020] master.HMaster: Client=hadoop//10.10.10.105 create '__10.10.10.113_Class_model0_gen0_1', {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'p', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 't', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
2015-07-08 20:43:36,681 INFO [MASTER_TABLE_OPERATIONS-HDPNN:16020-0] handler.CreateTableHandler: Create table __10.10.10.113_Class_model0_gen0_1
2015-07-08 20:43:36,716 INFO [RegionOpenAndInitThread->>__10.10.10.113_Class_model0_gen0_1-1] regionserver.HRegion: creating HRegion __10.10.10.113_Class_model0_gen0_1 HTD == '__10.10.10.113_Class_model0_gen0_1', {NAME => 'c', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'p', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 't', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS => 'FALSE', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'} RootDir = hdfs://10.10.10.105:9000/hbase/.tmp Table name == __10.10.10.113_Class_model0_gen0_1
2015-07-08 20:43:36,727 INFO [RegionOpenAndInitThread-__10.10.10.113_Class_model0_gen0_1-1] regionserver.HRegion: Closed __10.10.10.113_Class_model0_gen0_1,,1436388216644.65865aede9173944a90889efed79b0a7.
2015-07-08 20:43:36,731 INFO [MASTER_TABLE_OPERATIONS-HDPNN:16020-0] hbase.MetaTableAccessor: Added 1
2015-07-08 20:43:36,739 WARN [MASTER_TABLE_OPERATIONS-HDPNN:16020-0] zookeeper.ZKTableStateManager: Moving table __10.10.10.113_Class_model0_gen0_1 state from ENABLING to ENABLED
2015-07-08 20:43:36,742 INFO [MASTER_TABLE_OPERATIONS-HDPNN:16020-0] handler.CreateTableHandler: failed. null
