1

Trying to scan Bigtable which has 130 million rows using the HBase java client. After a few hours into the scan, it throws: 

INFO com.google.cloud.bigtable.grpc.async.AbstractRetryingOperation - Retrying failed call. Failure #1, got: Status{code=DEADLINE_EXCEEDED, description=Error while reading table '', cause=null} on channel 5.

There were a few such exceptions from which the retry recovered but eventually results in:

java.lang.IllegalStateException: Not started
    at com.google.common.base.Preconditions.checkState(Preconditions.java:507)
    at io.grpc.internal.ClientCallImpl.request(ClientCallImpl.java:367)
    at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34)
    at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22)
    at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.request(ForwardingClientCall.java:44)
    at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34)
    at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22)
    at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.request(ForwardingClientCall.java:44)
    at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34)
    at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22)
    at com.google.cloud.bigtable.grpc.io.RefreshingOAuth2CredentialsInterceptor$1.request(RefreshingOAuth2CredentialsInterceptor.java:146)
    at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34)
    at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22)
    at io.grpc.ForwardingClientCall$SimpleForwardingClientCall.request(ForwardingClientCall.java:44)
    at com.google.cloud.bigtable.grpc.io.Watchdog$WatchedCall.request(Watchdog.java:198)
    at io.grpc.PartialForwardingClientCall.request(PartialForwardingClientCall.java:34)
    at io.grpc.ForwardingClientCall.request(ForwardingClientCall.java:22)
    at com.google.cloud.bigtable.grpc.scanner.RetryingReadRowsOperation$CallToStreamObserverAdapter.request(RetryingReadRowsOperation.java:90)
    at com.google.cloud.bigtable.grpc.scanner.ResponseQueueReader.getNextMergedRow(ResponseQueueReader.java:109)
    at com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:78)
    at com.google.cloud.bigtable.grpc.scanner.ResumingStreamingResultScanner.next(ResumingStreamingResultScanner.java:35)
    at com.google.cloud.bigtable.hbase.adapters.read.BigtableResultScannerAdapter$1.next(BigtableResultScannerAdapter.java:66)
    at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:53)
Maxim
  • 4,075
  • 1
  • 14
  • 23
Srini
  • 53
  • 8
  • 1
    This seems like it might be a bug in the retry logic. Can you open an issue on https://github.com/GoogleCloudPlatform/cloud-bigtable-client/issues and assign it to igorbernstein2? Please include a list of your dependencies. Thanks! – Igor Bernstein Dec 07 '18 at 20:58
  • 1
    Thanks @IgorBernstein. Opened the issue but was unable to assign it to you. Here is the link - https://github.com/GoogleCloudPlatform/cloud-bigtable-client/issues/2020 – Srini Dec 08 '18 at 16:43
  • Thanks for opening the issue. I'll work on reproducing and fixing it this week – Igor Bernstein Dec 10 '18 at 20:12
  • Thanks for the update. There is one more minor bug which we opened as well - https://github.com/GoogleCloudPlatform/cloud-bigtable-client/issues/1976 – Srini Dec 11 '18 at 10:35

1 Answers1

0

Quick update: both of these issues were fixed in version 1.9.0

Igor Bernstein
  • 571
  • 2
  • 5