0

In python, I would like to:

  1. Obtain a filename from a user to a csv file
  2. convert the csv into a temporary table
  3. run user defined queries against a central database to add data to the table
  4. output to a new csv for the user to use.

I can handle most of these steps there are just a few things that I don't understand.

Is this possible using a sqlite script? Would python sqlite class be able to execute that script?

conn = sqlite3.connect(central_database, check_same_thread=False)
cur = self.conn.cursor()

#user defines csv file and other params via http form -> script

cur.execute(script)
csv = cur.fetchall()

I could be thinking about this all wrong. So here is my setup:

  1. python 3.6.3
  2. flask/jinja2 (javascript capable)
  3. sqlite 3.22.0
Josh Sharkey
  • 1,008
  • 9
  • 34
  • If this is a script that the user would run, I would recommend building your temporary table in Python using Pandas, do your modifications there, and then write that back out to a CSV. Is this an option? – Engineero Apr 20 '18 at 20:23
  • Yes. that should be easy in python. You can use the `csv` module to read / write the csv, and what would sqlite3 be if you couldn't query it. Except.... is it a sqlite database? If not, use a different python sql library. This is too broad for this site, so this will get closed, but continue on course. – tdelaney Apr 20 '18 at 20:24
  • pandas is fine but csv may be easier. So already we're bickering! – tdelaney Apr 20 '18 at 20:25
  • Assuming you can't use pandas, you could create a table and insert the csv data into the temp table. Once it's inserted, run the queries on your table. Fetch the data and put it in the csv. For inserting the data into SQL go here: https://stackoverflow.com/questions/21257899/writing-a-csv-file-into-sql-server-database-using-python – CrizR Apr 20 '18 at 20:27
  • You could likely do all of this by importing the csv to the database and doing the work in sql there. – tdelaney Apr 20 '18 at 20:29

1 Answers1

0

Haven't tested it, but you could try this:

conn = sqlite3.connect(central_database, check_same_thread=False)
cur = self.conn.cursor()

# Create table using first row of csv (column names)

#user defines csv file and other params via http form -> script
with open ('test.csv', 'r') as f:
    reader = csv.reader(f)
    columns = next(reader) 
    query = 'insert into MyTable({0}) values ({1})'
    query = query.format(','.join(columns), ','.join('?' * len(columns)))
    for data in reader:
        cur.execute(query, data)
    cur.commit()

cur.execute(script)
csv = cur.fetchall()

# Delete table 

Source: Writing a csv file into SQL Server database using python

CrizR
  • 688
  • 1
  • 6
  • 26
  • This sounds like something I could work into the code. Thank you Is this table in fact temporary? I want to ensure that after the query is done and the csv file created the table no longer exists in the database. – Josh Sharkey Apr 20 '18 at 20:53
  • My bad, I forgot to add the part where you create the table. It's as temporary as you want it to be. You'll need to create the table, and if you want, you can delete the table after the data has been exported. – CrizR Apr 20 '18 at 20:55
  • Could you specify the code to create the table? I'm assuming the columns variable will be important here since they are going to be used in the query – Josh Sharkey Apr 20 '18 at 21:19
  • I think im getting the jist of it. All manual query handling and saving. Thanks for the insight Chris! – Josh Sharkey Apr 20 '18 at 21:37
  • You’ll need to build the query string using the column names from the first row of the CSV. Something like: cur.execute(“drop table if exists temp_table”) followed by cur.execute(“create table temp_table ( column_name type, ...) – CrizR Apr 20 '18 at 22:23
  • And to build the query create a dictionary where the column name maps to the type. – CrizR Apr 20 '18 at 22:24
  • Chris: I'm coming to the realization that this may be a bit slow if the CSV file is very large. Is there another approach that uses the '.import {filename}' sqlite3 function? – Josh Sharkey Apr 24 '18 at 04:04