Unable to load data into mysql using python

Question

I was not able to load symbol column into mysql. File name is BAJFINANCE.NS.csv

Date,Open,High,Low,Close,Adj Close,Volume   
2002-07-01,5.239740,5.540930,5.239740,5.412680,0.063393,21923  
2002-07-02,5.523440,5.537040,5.421420,5.443770,0.063757,61045  
2002-07-03,5.505950,5.545790,5.328150,5.466120,0.064019,34161

in mysql am getting

timestamp,open,high,low,close,adj_close,volume,symbol  
2002-07-01,5.23974,5.54093,5.23974,5.41268,0.063393,21923,NULL  
2002-07-02,5.52344,5.53704,5.42142,5.44377,0.063757,61045,NULL  
2002-07-03,5.50595,5.54579,5.32815,5.46612,0.064019,34161,NULL

How to include symbol in mysql

import mysql.connector
config = {
'user':'root',
'password':'root',
'host':'127.0.0.1',
'database':'masterfiles'
 }
 conn= mysql.connector.connect(**config)
 c=conn.cursor()

def insertRows(fileName,c):
    delimiter=r','
    file=fileName.split("/")[-1]
    symbol = file[:-7]
    if file.startswith("BAJ"):
        c.execute("""Load data local infile %s into table adjprice fields terminated by %s ignore 1 lines 
                 (timestamp,open,high,low,close,adj_close,volume,@symbol)""",(fileName,delimiter))

localExtractFilePath="/Users/dalal"

import os

for file in os.listdir(localExtractFilePath):
    if file.endswith("csv"):
        insertRows(localExtractFilePath+"/"+file,c)
        print "Loaded file "+file+" into database"
        conn.commit()
c.close()
conn.close()

score 1 · Accepted Answer · answered Jun 17 '18 at 13:25

I think this will do what you want:

       c.execute("""
Load data local infile %s
    into table adjprice
    fields terminated by %s
    ignore 1 lines 
    (timestamp, open, high, low, close, adj_close, volume)
    set symbol = %s
""", (fileName, delimiter, symbol))

I would recommend using query parameters rather than munging the string. However, I'm not 100% sure if the filename and fields components can be passed in as parameters.

score 0 · Answer 2 · answered Jun 18 '18 at 02:38

I am not sure how committed you are to using the imported modules that you have in your question but I would approach this problem by using Pandas and SQLalchemy for the ease of it all.

Here is how I would approach your problem:

import os
import pandas as pd
from sqlalchemy import create_engine

localExtractFilePath="/Users/dalal"

config = {
'user':'root',
'password':'root',
'host':'127.0.0.1',
'database':'masterfiles'
 }


def main():
    engine = create_engine('mysql://{}:{}r@{}/{}'.format(config['user'],config['password'],config['host'],config['database']))

    for file in os.listdir(localExtractFilePath):
        if file.endswith("csv"):
            print('{} file has been found. Loading to Pandas DF'.format(file))
            #Load to a Pandas DataFrame Object (Documentation: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html)
            df = pd.read_csv(os.path.abspath(file))
            df['symbol'] = os.path.splitext(file)[0][:-7]
            print('Loading to DB....')
            #Upload DF object to SQL (Documentation: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html)
            df.to_sql(con=engine, index=False, if_exists='append')
            print('Done, on to the next file!')

if __name__ == '__main__':
    main()

Hope you find this helps!

Unable to load data into mysql using python

2 Answers2