Pandas DataFrame.to_csv() doesn't output all columns

Question

I have a large dataframe with ~450 columns. I need to export it to csv, and for that I use the next command:

df.to_csv('output.csv', sep='\t', header=False, index=False, na_rep='null')

The resulting file is missing several of the trailing columns. I tried to update pandas display options, but that didn't solve the problem:

pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', -1)
pd.set_option('display.width', None)

Any help is appreciated.

Background: I need to export data from one DB table to the other, adding one more column on the fly. I export data to csv, create a dataframe from it, add a column, and then need to export it to csv again to import into the resulting table.

Why can't you copy the dataframe and a column to the new df? — Rahul Agarwal, Dec 06 '18 at 08:31
Have you tried setting
to check which columns are missing? alternatively, try using and see if issue persists. — ParvBanks, Dec 06 '18 at 08:32
Maybe it goes wrong when you're importing the file from DB to a df? Or do you have the right amount of columns when you do the import and it only goes wrong with the final export? — Sander van den Oord, Dec 06 '18 at 08:49

score 1 · Accepted Answer · answered Apr 15 '19 at 14:54

1

As it turned out, the original data had a quotation mark somewhere. This answer to another question helped me sort it out. I resolved the problem with:

pd.read_csv(file_obj, sep='\t', quoting=csv.QUOTE_NONE)

answered Apr 15 '19 at 14:54

NothisIm

53
1
6

Pandas DataFrame.to_csv() doesn't output all columns

1 Answers1