0

Iam trying to concatenate multiple csv but the output file has all the data in a single row. How to add a new line character while giving a output. Here is a sample code of mine

from os import chdir
from glob import glob
import pandas as pdlib

# Produce a single CSV after combining all files
def produceOneCSV(list_of_files, file_out):
   # Consolidate all CSV files into one object
   result_obj = pdlib.concat([pdlib.read_csv(file) for file in list_of_files], sort=False)
   # Convert the above object into a csv file and export

   result_obj.to_csv(file_out, index=False, encoding="utf-8")

# Move to the path that holds our CSV files
csv_file_path = 'Path_for_all_the_csv/'
chdir(csv_file_path)

# List all CSV files in the working dir
file_pattern = ".csv"
list_of_files = [file for file in glob('*.csv')]
print(list_of_files)

file_out = "ConsolidateOutput.csv"
produceOneCSV(list_of_files, file_out)

I followed the document https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html. But didnt worked for me. Is there any change in document?

Ankzious
  • 311
  • 3
  • 18
  • Does this answer your question? [Import multiple csv files into pandas and concatenate into one DataFrame](https://stackoverflow.com/questions/20906474/import-multiple-csv-files-into-pandas-and-concatenate-into-one-dataframe) – Ron Jun 02 '20 at 12:29
  • If you want to create a consolidated csv, why are you involving pandas in this? Can't you just read each file one by one and write it out in to a larger file? – najeem Jun 02 '20 at 12:32

1 Answers1

2

Do you need pandas to concatenate a bunch of csvs? If you're using linux, you can do it with one command.

$ cat *.csv > consolidated.csv

And if you want to use python

with open("consolidated.json", "w") as f:
    for ff in glob.glob("*.csv"):
        with open(ff) as fff:
            f.write(fff.read())
najeem
  • 1,841
  • 13
  • 29