0

I'm currently develop a Jupyter notebook with Python code, in Mac OSX Mojave 10.14. I'm using Psycopg2 to get data from PostgreSQL to Python Dataframe.

The problem is the code works fine in Win 10, in two ways, but I not get data from the same query in Mac OSX.

I don't get any error, I just get an empty Dataframe. Nothing more. No error, no warning.

Anyone have been in the same trouble?

The used code is below:

First option:

cursor = conn.cursor()
query = """select * from table"""

cursor.execute(query)
data = data.append(cursor.fetchall())
data.columns = [desc[0] for desc in cursor.description]

Second option:

data= pd.DataFrame()

for chunk in pd.read_sql("""select * from table""", con=conn, chunksize=5000):
    data= data.append(chunk)

I expect a Dataframe with the data from PostgreSQL.

cwalvoort
  • 1,851
  • 1
  • 18
  • 19
Jorge Martinez
  • 1
  • 1
  • What is your `data`? It it's a Python list, `data.append` does not return the updated list, it mutates the list (`data`) in place and returns None. So `data = data.append(...)` effectively sets `data` to `None`. It also will append all the rows in a single element; you likely need `deta.extend()`. – 9000 May 29 '19 at 16:12
  • @Zoe `data` is a new DataFrame created before the query, not a Python list. Like I said, in Win 10 it run perfect, but I have problems in Mac Osx, with the same tools and environments. When I tried to execute `data.extend()` I get the error `AttributeError: 'DataFrame' object has no attribute 'extend'` – Jorge Martinez May 29 '19 at 16:46
  • Please post all problems/errors/undesired results in Mac OSX. Simply stating I do not get data is not helpful. Secondly [Never call DataFrame.append or pd.concat inside a for-loop. It leads to quadratic copying.](https://stackoverflow.com/a/36489724/1422451). – Parfait May 29 '19 at 17:03
  • @Parfait don't get any error, I just get an empty DataFrame. I don't have a nothing to show, cause there is no exist an error. I checked the code several times. Secondly, thanks for the link. I gonna change the way to code that. Now I know is inefficient. – Jorge Martinez May 29 '19 at 20:49
  • Are both scripts pointing to same Postgres backend? Maybe it is a database copy with no data? Both Win and Mac on same server or remote? I doubt psycop2 behaves drastically different by OS. – Parfait May 29 '19 at 21:54

1 Answers1

0

The problem was that I had a '%%time' statement before the query, to get the execute time of the cells. Funny. It seems in Mac OSX doesn't work with a piece of code below.

Jorge Martinez
  • 1
  • 1
  • The problem is the bug in ipython 7.4 if you downgrade to 7.3 or upgrade to 7.5 you can use %%time again. – rpanai May 29 '19 at 22:51