15

I ran a following simple script:

import sys
import pandas as pd
df = pd.DataFrame([[1,2,3], [4,5,6]])
df.to_csv(sys.stdout)

Expected Output (as standard output)

,0,1,2
0,1,2,3
1,4,5,6

However, I got no outputs on standard output by the program under Python 2.7.15 on macOS 10.12.6. Instead, it generated a file named <stdout> which contains the expected output.

Interestingly, on the same OS, Python 3.6.5 could show the result as standard output without any problems, and virtualenved Python 3.6.5 could not show it (and generated the <stdout> file).

Does anyone identify the cause of this result? The version of Pandas is 0.23.1.

kiichi
  • 153
  • 1
  • 6

2 Answers2

11

Depending on what you are ultimately trying to do this workaround might or might not be useful:

In [85]: from io import StringIO

In [86]: output = StringIO()

In [87]: df.to_csv(output)

In [88]: print(output.getvalue())
,0,1,2
0,1,2,3
1,4,5,6

(Python 3.6)

For Python 2.7 replace the one line above with:

from StringIO import StringIO

UPDATE:

Actually, I think this might be the 'proper' way to do it:

In [3]: from io import StringIO

In [4]: output = StringIO()

In [5]: df.to_csv(output)

In [6]: output.seek(0)
Out[6]: 0

In [7]: print(output.read())
,0,1,2
0,1,2,3
1,4,5,6
Bill
  • 10,323
  • 10
  • 62
  • 85
  • FutureWarning: pyarrow.open_stream is deprecated as of 0.17.0, please use pyarrow.ipc.open_stream instead. – Akshay Hazari Aug 18 '23 at 02:51
  • @AkshayHazari can you please explain your comment. I don't see the connection to this answer. – Bill Aug 18 '23 at 05:46
  • When spark dataframe is converted to Pandas and then to_csv , it doesn't give the results but only this warning. But when I use `sys.stdout` instead of `StringIO()` it gave the required output along with the above warning. Then I also turned the warnings off using `warnings.filterwarnings('ignore')` to remove the above warning and only get the result – Akshay Hazari Aug 18 '23 at 16:28
5

Looks like this is a known issue:

https://github.com/pandas-dev/pandas/issues/21561

https://github.com/pandas-dev/pandas/pull/21478

cosmic_inquiry
  • 2,557
  • 11
  • 23