2

On Azure, I have database (delta) on ADLS (gen2) which can be accessed by Databricks.

I am doing a performance test from a client application which usage Simba JDBC driver to connect to data bricks and get data from database. When I do a continuous load of around 6 millions of data, the process breaks after 1 millions of data due to an error from databricks. I checked the cluster didn’t went down in this time but my client was not able to connect and throw below error.

My Databricks Cluster config is:

Worker Type : Standard_DS4_v2 - 28 GB, 8 cores

Driver Type:Standard_DS5_v2 - 56 gb , 16 core

I also tried to increase Driver Type to (Driver Type: Standard_DS5_v2 - 56 gb , 16 cores) but that failed

Error:

[Access(81040)00-ids.databricks] ERROR <date time> com.denodo.vdb.engine.wrapper.raw.jdbc.adapter.SQLPlugin [] - Error validating connection
java.sql.SQLException: [Simba][SparkJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: com.simba.spark.jdbc42.internal.apache.http.NoHttpResponseException: <Databricks URL> failed to respond.
        at com.simba.spark.hivecommon.api.HS2Client.handleTTransportException(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.api.HS2Client.executeStatementInternal(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.api.HS2Client.executeStatement(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeQuery(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.<init>(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.dataengine.HiveJDBCDataEngine.prepare(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.jdbc.common.SStatement.executeNoParams(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.jdbc.common.SStatement.execute(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.jdbc42.Hive42Statement.execute(Unknown Source) ~[SparkJDBC42.jar:?]
        at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194) ~[commons-dbcp2.jar:2.7.0]
        at com.denodo.vdb.engine.wrapper.raw.jdbc.adapter.SQLPlugin.executeValidationQuery(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.connection.jdbc.JDBCConnectionFactory.a3(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.connection.jdbc.JDBCConnectionFactory.validateConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at org.apache.commons.dbcp2.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:648) [commons-dbcp2.jar:2.7.0]
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:472) [commons-pool2.jar:2.7.0]
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) [commons-pool2.jar:2.7.0]
        at com.denodo.vdb.misc.connection.jdbc.JDBCConnectionPool.borrowObject(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.connection.jdbc.JDBCConnectionPool.getCredentialedConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.datasource.credentialed.CredentialedDatasource.getCredentialedConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.datasource.JDBCDataSource.c3(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.datasource.JDBCDataSource.getConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.datasource.JDBCXADataSource.getNonXAConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.datasource.JDBCXADataSource.getNonXAConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.misc.connection.jdbc.JDBCRoute.getConnection(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.engine.wrapper.raw.jdbc.JDBCAccess.doRun(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.engine.wrapper.RawAccess.run(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.engine.thread.WorkWrapper.a3(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.engine.thread.ReusableThread.d3(Unknown Source) [denodo-vdp-server.jar:8.0.0]
        at com.denodo.vdb.engine.thread.ReusableThread.run(Unknown Source) [denodo-vdp-server.jar:8.0.0]
Caused by: com.simba.spark.support.exceptions.ErrorException: [Simba][SparkJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: com.simba.spark.jdbc42.internal.apache.http.NoHttpResponseException: <Databricks URL> failed to respond.
        
Caused by: com.simba.spark.jdbc42.internal.apache.thrift.transport.TTransportException: com.simba.spark.jdbc42.internal.apache.http.NoHttpResponseException: <Databricks URL> failed to respond
        at com.simba.spark.hivecommon.api.TETHttpClient.flushUsingHttpClient(Unknown Source) ~[SparkJDBC42.jar:?]
        at com.simba.spark.hivecommon.api.TETHttpClient.flush(Unknown Source) ~[SparkJDBC42.jar:?]
AmitG
  • 519
  • 6
  • 19
  • You can find the same issue information here : https://stackoverflow.com/questions/64176301/connecting-odbc-to-azuredatabricks-using-simba-driver – Karthikeyan Rasipalay Durairaj Dec 03 '21 at 00:15
  • This is not the same problem. My token is correct and valid for 1 hr. I am able to get the data for sometime but later sometime after 1million it breaks. It can be 110k or 150k. – AmitG Dec 06 '21 at 21:52
  • Hey @AmitG, I'm encountering a similar problem. Were you able to resolve this? – Kishore Bandi Mar 09 '23 at 14:42

0 Answers0