Can not read large data from phoenix table

Question

Hi All i am getting below error message while running phoenix count query on a large table.

0: jdbc:phoenix:hadoopm1:2181> select Count(*) from PJM_DATASET;
+------------+
|  COUNT(1)  |
+------------+

java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Jan 09 02:18:10 CST 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=62365: row '' on table 'PJM_DATASET' at region=PJM_DATASET,,1420633295836.4394a3aa2721f87f3e6216d20ebeec44., hostname=hadoopctrl,60020,1420790733247, seqNum=27753

    at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
    at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
    at sqlline.SqlLine.print(SqlLine.java:1735)
    at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
    at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
    at sqlline.SqlLine.dispatch(SqlLine.java:821)
    at sqlline.SqlLine.begin(SqlLine.java:699)
    at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
    at sqlline.SqlLine.main(SqlLine.java:424)
0: jdbc:phoenix:hadoopm1:2181>

please help.

i have two master and 1 region server at my cluster, ambari service check and status showing that every thing is working fine — user3683741, Jan 09 '15 at 09:51
I have the exact same problem. The size of my table is 1.5 billion rows. — Zoltán, May 06 '15 at 12:26
@shaileshgupta This is probably irrelevant at this point, but I've posted a new answer below. Perhaps it'll help. — Malt, Dec 03 '18 at 14:59
@Zoltán This is probably irrelevant at this point, but I've posted a new answer below. Perhaps it'll help. — Malt, Dec 03 '18 at 15:00

score 2 · Answer 1 · answered Nov 25 '15 at 17:20

You need to increase the below hbase configuration property to a higher value in hbase server and client configurations, default value is 1 min(60000 Milli Sec)

  <property>
    <name>hbase.rpc.timeout</name>
    <value>600000</value>
  </property>

The most important thing is that the below command should return the correct hbase config directory where hbase-site.xml file contains the above property. If the value of hbase_conf_path is empty or ., Execute the command export HADOOP_CONF_DIR=<correct_hbase_dir>; for pointing to correct hbase configuration before executing the command sqlline.py.

phoenix_utils.py  | grep hbase_conf_path

Changing only hbase.rpc.timeout doesn't help. Need to set hbase.client.scanner.timeout.period as well. See answer from @Malt below — vmaroli, Jan 16 '19 at 04:46

score 2 · Answer 2 · answered Dec 03 '18 at 14:53

(Three years after the original question)

We ran into very similar exceptions today:

java.net.SocketTimeoutException: callTimeout=60000, callDuration=60101: row '' on table '****' at region=****, hostname=wn1-bghbas.y1ao3ht4safu3e4em4fhuri0eh.ax.internal.cloudapp.net,16020,1543844388080, seqNum=568644
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Mon Dec 03 14:01:45 UTC 2018, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60101: row '' on table '****' at region=****, hostname=****, seqNum=568644

       at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:271)
       at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:210)
       at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
       at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
       at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:327)
       at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:413)
       at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371)
       at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:55)
       ... 11 more

After tracking down the exception's call trace, we figured out that the right configuration variable is hbase.client.scanner.timeout.period. Which can be found in class org.apache.hadoop.hbase.HConstants under the name HBASE_CLIENT_SCANNER_TIMEOUT_PERIOD.

Note that if you're passing a value for hbase.client.scanner.timeout.period as a property to PhoenixDriver using raw JDBC, the value has to be String, not int.

Something like:

Properties prop = new Properties();
prop.put("user", "***");
prop.put("password", "***");
prop.put(HConstants.HBASE_CLIENT_SCANNER_TIMEOUT_PERIOD, "800000");
Connection connection = DriverManager.getConnection("jdbc:phoenix:10.0.0.1:2181:/hbase-unsecure", prop);

For the record, we're using Phoenix 4.7 on Hbase 1.1

We had the same issue and setting hbase.client.scanner.timeout.period has solved the issue in addition to setting hbase.rpc.timeout and phoenix.query.timeoutMs — vmaroli, Jan 16 '19 at 04:41

score 0 · Answer 3 · answered May 13 '15 at 07:55

0

callTimeout=60000, callDuration=62365.

Looks like your call duration is longer than 10 minutes, which is the timeout duration. Try setting the "hbase.master.catalog.timeout" in the hbase-default.xml - (or your custom )config file to a higher value

answered May 13 '15 at 07:55

Alexis Peters

1,583
1
10
17

1

Thanks for the answer, but the reported timeout is `60000 ms`, which is 1 minute, not 10 - whereas the default value of `hbase.master.catalog.timeout` is 10 minutes, so I don't believe this is causing the `SocketTimeoutException` – Zoltán May 18 '15 at 08:50

score 0 · Answer 4 · answered Nov 27 '21 at 13:09

0

Yes, we add the same property while running the command and get resolved this problem hbase.client.scanner.timeout.period=600000

Hbase 1.2

answered Nov 27 '21 at 13:09

Pavan Gomladu

66
1
8

Can not read large data from phoenix table

4 Answers4