I'm new to python multiprocessing. I'm trying to read excel file with enormous file size. When reading big files it takes a long time and I just found out about Multiprocessing It is the technology that allows your program to run in parallel by using multiple CPU cores at the same time. It is used to significantly speed up your program I'm trying to use it but when I read it via Pool, it returns error no such file or directory but without pool (just call the function with the parameter) it doesn't raise error.
import os
import pandas as pd
import multiprocessing as mp
def read_csv(filename):
return pd.read_csv(filename)
def main():
myfile = "C:\\excel_processor\\1_wht_escrow_verified_id -1.csv"
pool = mp.Pool(2)
mydf = pool.map(read_csv, myfile)
if __name__ == '__main__':
main()
