While calling function with pool, It shows the variable is not defined which is there in main (global)

Question

May be my question looks simple (or the Bug might be minor) but I could not be able find the bug, really I struggled a lot to figure out the issue.

I've created a Framework to Extract the Data from Salesforce with Simple Salesforce package, but I've encountered with the bug when I'm using multiprocessing.

My code is pretty much straight forward but tedious. I don't want to paste entire code here, So here is my Code from my GitHub.

Issue:

When I'm calling this Extract Data function with Pool, the variable which is there in the __name__ == '__main__' is not working.

In Other words I'm getting, NameError: name 'SFAPI' is not defined - But It's there in the main as Global Variable and It's working without pool (A single call).

Execution Example:

python "E:\Documents\myPy\SF Project\sf_extraction.py" -pr data_extraction -tn Opportunity Account

Small Snippet from my code, where I'm getting issues:

def ExtractData(table_name):
    logging.info('Extract Data for Table Name: ' + table_name + ' at ' + getCurrDatetime())
    try:
        rec_count = getRecordCount(table_name)
        print(rec_count)
        if int(rec_count) == 0:
            logging.info('There is no data to Extract for {}'.format(table_name))
        else:
            soql = SFAPI.CreateSOQL(table_name)
            data = SFAPI.ExecuteSOQL(soql, is_count=0)
            extract_file_nm = table_name + '_' + db_name + '_'  + sc_name + '_' + curr_datetime + '.csv'
            print(data)
            print(type(data))
            extract_file = os.path.expanduser(os.path.join(script_path,extract_file_nm))
            data.to_csv(extract_file, index=False)
            logging.info('Data has been extrcated as {} at {}'.format(extract_file, getCurrDatetime()))

    except Exception as e:
        logging.info('Error in Extraction')
        err_msg = "FATAL_ERROR: In the ExtractData Function : {0}\n\n{1}".format(e, traceback.format_exc())
        raise Exception(str(err_msg))

Place or Snippet from where I'm calling this:

if __name__ == '__main__':
    try:
        SFAPI = SalesforceAPICall(username=config['username'],
                            password=config['password'],
                            security_token=config['sf_token'],
                            )

        if len(table_name) != 0 and 'data_extraction' in process_nm:
            try: 
                if len(table_name) == 1:
                    print(table_name[0])
                    ExtractData(table_name[0])
                if type(table_name) == list and len(table_name) > 1:
                    #p = Pool(processes=int(processes))
                    print('Calling Pool : ' + str(os.cpu_count()))
                    #out = p.map(ExtractData, table_name)
                    #p.close()
                    #p.join()
                    p = Pool()
                    print(table_name)
                    x = p.map(ExtractData, table_name)
                    x.get()
                    p.close()
                    p.join()
            except Exception as e:
                if len(table_name) > 1:
                    p.terminate()
                    p.join()
                logging.error("Process Failed - " + str(e))
    except Exception as e:
        chk_err('FATAL_ERROR: ' + " from main exception : {0}\n\n{1}".format(e, traceback.format_exc()))

You can very well refer my code in GitHub if it looks clumsy or if you feel this is not enough amount of information to fix.

Again it might be a small Bug, Hope You Understand what I'm trying to convey !!! Thanks in Advance !!!

Regards, Parvathirajan Natarajan

While calling function with pool, It shows the variable is not defined which is there in main (global)

0 Answers0