Issue in Azure deployed Databricks using Python's read_csv on small csv. Py4j: Error while obtaining a new communication channel

Question

I am using an Azure Databricks Service and running a Python notebook.

I have mounted an Azure Data Lake Storage Gen1 to the Databricks and running the command:

import pandas as pd

meta = pd.read_csv(f"{PATH}/meta_day.csv") # PATH is path in the datalake

This meta_day.csv is fairly small and sits around 1kB. Most of the time the command runs fine. However now and again I get ConnectException: Connection refused (Connection refused), which is tracked back to:

21/09/20 09:34:15 INFO PythonDriverWrapper: Repl ReplId-6c64d-99db0-93ca2-4 got an exception during execution
py4j.Py4JException: Error while obtaining a new communication channel
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:257)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:377)
    at py4j.CallbackClient.sendCommand(CallbackClient.java:356)
    at py4j.reflection.PythonProxyHandler.invoke(PythonProxyHandler.java:106)
    at com.sun.proxy.$Proxy412.setReplClassLoader(Unknown Source)
    at com.databricks.backend.daemon.driver.PythonDriverLocal.$anonfun$setReplClassLoader$1(PythonDriverLocal.scala:336)
    at com.databricks.backend.daemon.driver.PythonDriverLocal.$anonfun$setReplClassLoader$1$adapted(PythonDriverLocal.scala:336)
    at com.databricks.backend.daemon.driver.PythonDriverLocal.withInterpLock(PythonDriverLocal.scala:775)
    at com.databricks.backend.daemon.driver.PythonDriverLocal.setReplClassLoader(PythonDriverLocal.scala:336)
    at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:601)
    at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:689)
    at scala.util.Try$.apply(Try.scala:213)
    at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:681)
    at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:522)
    at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:634)
    at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:427)
    at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:370)
    at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:221)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at java.net.Socket.connect(Socket.java:556)
    at java.net.Socket.<init>(Socket.java:452)
    at java.net.Socket.<init>(Socket.java:262)
    at javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:277)
    at py4j.CallbackConnection.start(CallbackConnection.java:226)
    at py4j.CallbackClient.getConnection(CallbackClient.java:238)
    at py4j.CallbackClient.getConnectionLock(CallbackClient.java:250)
    ... 18 more

Example of a file that breaks it (first line if the header):

A,B,C,D,E
2612575e-7177-4e1c-b304-eef51fbe5c05,2021-04-07T19:36:37.84+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,-1,False
f764f896-f01d-4d0e-b781-17ed83758bab,2021-04-08T14:28:26.727+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,-1,False
05f23e66-1b18-478e-9891-ccc13f6cb135,2021-04-08T14:24:42.004+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,1,False
c3975364-8331-46ed-b9af-529a9c321da9,2021-04-07T19:36:33.537+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,1,False

I have managed to narrow down that if I remove one character from the first element 2612575e-7177-4e1c-b304-eef51fbe5c05, then there is not a problem...?!

Additional information

File that intermittently works, depending on the content:

A, B, C, D, E
2612575e-7177-4e1c-b304-eef51fbe5c05,2021-04-07T19:36:37.84+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,-1,False
f764f896-f01d-4d0e-b781-17ed83758bab,2021-04-08T14:28:26.727+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,-1,False
05f23e66-1b18-478e-9891-ccc13f6cb135,2021-04-08T14:24:42.004+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,1,False
c3975364-8331-46ed-b9af-529a9c321da9,2021-04-07T19:36:33.537+00:00,e80168ed-ebbb-4fb3-9124-5d11501cce74,1,False

Accompanying file that also works, regardless of the content:

A, B, C, D, E
2021-04-07 08:40:53.290, d7d06da3-ed99-4f16-904f-046d296b4ef1, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 764.93937254902
2021-04-07 08:40:53.290, d7d06da3-ed99-4f16-904f-046d296b4ef1, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1225.21
2021-04-07 08:40:53.290, d7d06da3-ed99-4f16-904f-046d296b4ef1, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.54
2021-04-07 08:40:53.290, d7d06da3-ed99-4f16-904f-046d296b4ef1, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 918.52
2021-04-07 04:57:30.615, 21628b55-e599-4c9f-9649-72a9dee6bc56, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 774.613565891473
2021-04-07 04:57:30.615, 21628b55-e599-4c9f-9649-72a9dee6bc56, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1261.82
2021-04-07 04:57:30.615, 21628b55-e599-4c9f-9649-72a9dee6bc56, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.57
2021-04-07 04:57:30.615, 21628b55-e599-4c9f-9649-72a9dee6bc56, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 903.27
2021-04-07 08:39:08.045, 7d867cd9-c2e3-41b1-90e4-0fd298ee6097, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 725.701106719368
2021-04-07 08:39:08.045, 7d867cd9-c2e3-41b1-90e4-0fd298ee6097, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1217.58
2021-04-07 08:39:08.045, 7d867cd9-c2e3-41b1-90e4-0fd298ee6097, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.52
2021-04-07 08:39:08.045, 7d867cd9-c2e3-41b1-90e4-0fd298ee6097, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 848.34
2021-04-07 04:32:17.967, d702d78c-7365-461f-a1ae-01229f379756, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 759.733515625
2021-04-07 04:32:17.967, d702d78c-7365-461f-a1ae-01229f379756, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1264.88
2021-04-07 04:32:17.967, d702d78c-7365-461f-a1ae-01229f379756, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.55
2021-04-07 04:32:17.967, d702d78c-7365-461f-a1ae-01229f379756, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 904.79
2021-04-07 04:34:03.641, f4b498fe-c02a-4868-8c80-f4072c0c4ad5, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 778.096850393701
2021-04-07 04:34:03.641, f4b498fe-c02a-4868-8c80-f4072c0c4ad5, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1272.51
2021-04-07 04:34:03.641, f4b498fe-c02a-4868-8c80-f4072c0c4ad5, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.53
2021-04-07 04:34:03.641, f4b498fe-c02a-4868-8c80-f4072c0c4ad5, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 924.63
2021-04-07 08:40:53.290, 353a9f1a-2e16-4f0f-9d7c-a2813fda7d0c, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 764.93937254902
2021-04-07 08:40:53.290, 353a9f1a-2e16-4f0f-9d7c-a2813fda7d0c, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1225.21
2021-04-07 08:40:53.290, 353a9f1a-2e16-4f0f-9d7c-a2813fda7d0c, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.54
2021-04-07 08:40:53.290, 353a9f1a-2e16-4f0f-9d7c-a2813fda7d0c, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 918.52
2021-04-07 04:57:24.570, d5f7251d-1b15-42b5-84ab-749b5ec1a8c2, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 772.124076923077
2021-04-07 04:57:24.570, d5f7251d-1b15-42b5-84ab-749b5ec1a8c2, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1264.88
2021-04-07 04:57:24.570, d5f7251d-1b15-42b5-84ab-749b5ec1a8c2, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.59
2021-04-07 04:57:24.570, d5f7251d-1b15-42b5-84ab-749b5ec1a8c2, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 903.27
2021-04-07 08:39:08.045, 6015e811-6781-4a82-bfcc-6bf4dbbe2d89, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 725.701106719368
2021-04-07 08:39:08.045, 6015e811-6781-4a82-bfcc-6bf4dbbe2d89, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1217.58
2021-04-07 08:39:08.045, 6015e811-6781-4a82-bfcc-6bf4dbbe2d89, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.52
2021-04-07 08:39:08.045, 6015e811-6781-4a82-bfcc-6bf4dbbe2d89, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 848.34
2021-04-07 16:33:00.807, 107709d1-7a97-493f-a627-2e09fc7b8e87, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 711.058492063492
2021-04-07 16:33:00.807, 107709d1-7a97-493f-a627-2e09fc7b8e87, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1211.47
2021-04-07 16:33:00.807, 107709d1-7a97-493f-a627-2e09fc7b8e87, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.51
2021-04-07 16:33:00.807, 107709d1-7a97-493f-a627-2e09fc7b8e87, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 830.03
2021-04-07 16:25:21.253, 9aae54d9-ad78-4239-be61-0f2abdbf47a7, e80168ed-ebbb-4fb3-9124-5d11501cce74, a, 775.449182879377
2021-04-07 16:25:21.253, 9aae54d9-ad78-4239-be61-0f2abdbf47a7, e80168ed-ebbb-4fb3-9124-5d11501cce74, b, 1234.36
2021-04-07 16:25:21.253, 9aae54d9-ad78-4239-be61-0f2abdbf47a7, e80168ed-ebbb-4fb3-9124-5d11501cce74, c, 2.56
2021-04-07 16:25:21.253, 9aae54d9-ad78-4239-be61-0f2abdbf47a7, e80168ed-ebbb-4fb3-9124-5d11501cce74, d, 900.21

Does [this](https://stackoverflow.com/questions/6876266/java-net-connectexception-connection-refused) help? — The Singularity, Sep 20 '21 at 10:30
I am confident there is nothing wrong with the connection setup, it seems heavily linked to the content of the file i.e. when I repeatedly try the example file the error appears. When I make the alteration, it works fine. — AmyChodorowski, Sep 20 '21 at 10:46
Yes it works fine with other files, it appears to be the format of this particular file that the error appears. — AmyChodorowski, Sep 22 '21 at 09:59
Any details that make this file different from the files that work — The Singularity, Sep 22 '21 at 10:02

Issue in Azure deployed Databricks using Python's read_csv on small csv. Py4j: Error while obtaining a new communication channel

0 Answers0