0

The code is the following:

import openpyxl
import pandas as pd

def copy_csv_to_xlsx(shipment_all, shipment_truck, stockflow_all, stockflow_truck, xlsx_file):

    shipment_all_df = pd.read_csv(shipment_all)
    shipment_truck_df = pd.read_csv(shipment_truck)
    stockflow_all_df = pd.read_csv(stockflow_all)
    stockflow_truck_df = pd.read_csv(stockflow_truck)
    
    
    wb = openpyxl.load_workbook(xlsx_file)
    
    
    ws = wb["0. Shipment"]
    ws.append(shipment_all_df.columns)
    for row in shipment_all_df.itertuples(index=False):
        ws.append(row)
    
    ws = wb["0. Truck Shipment"]
    ws.append(shipment_shave_care_df.columns)
    for row in shipment_shave_care_df.itertuples(index=False):
        ws.append(row)
        
    ws = wb["0. SF"]
    ws.append(stockflow_all_df.columns)
    for row in stockflow_all_df.itertuples(index=False):
        ws.append(row)
        
    ws = wb["0. Truck SF"]
    ws.append(stockflow_truck_df.columns)
    for row in stockflow_truck_df.itertuples(index=False):
        ws.append(row)
    
    
    wb.save(xlsx_file)
    

copy_csv_to_xlsx("Shipment All.csv", "Shipment Trucks.csv", "Stockflow All.csv", "Stockflow Truck.csv", "Business Split Data.xlsx")

It really does consume 8GB of memory and never finishes. What can be the reason and how do you think I can fix this problem? Thanks in advance!

shippy
  • 23
  • 6
  • Ive had lots of problems previously opening large excel files in python, It loads the entire spreadsheet in memory before doing anything and also uses a bunch of memory when parsing the files. Have you tried reading one csv/excel file at time, rather than load all of them at the start of your function – Tom McLean Feb 10 '23 at 10:36
  • And try using the chunksize option https://stackoverflow.com/questions/45532711/pandas-read-csv-method-is-using-too-much-ram – Tom McLean Feb 10 '23 at 10:37
  • See simpler approaches which do not involve row-by-row [here](https://stackoverflow.com/questions/17684610/python-convert-csv-to-xlsx) – user19077881 Feb 10 '23 at 10:56

0 Answers0