1

I'm new to Python/Pandas and SQLite. After browsing other stackoverflow help, I've created a few large SQLite databases by uploading csv files (3GB+ each). Want to do a join and the way I saw on here appears to not allow this. Is there a way to do it in this construct? Should I have uploaded multiple tables into a single SQLite database? Thanks in advance.

Assume my databases are

foo.db  
bar.db

For a single query of a single database I have the following (works):

import pandas as pd
from sqlalchemy import create_engine
import datetime as dt
from IPython.display import display


disk_engine = create_engine('sqlite:///foo.db')

df = pd.read_sql_query('SELECT * FROM data', disk_engine)
nonegiven72
  • 427
  • 8
  • 19
  • first of all, i would strongly recommend you to use RDBMS like (MySQL, SQLite, Oracle, etc.) for joining such amount of data - RDMSs are designed for that. And of course you want to have both tables in one DB instance, so you can join them. I personally would choose MySQL (more powerful compared to SQLite and still free) for your task – MaxU - stand with Ukraine Mar 02 '16 at 13:32
  • I don't think you can SQL join across different files. The query engine for sqlite only looks at one database at a time – OneCricketeer Mar 02 '16 at 13:35
  • Thanks all. I will go ahead and upload the csv's to a single SQLite database. I don't think I have the rights to install a MySQL database (or others) but will check. – nonegiven72 Mar 02 '16 at 13:46
  • You can run queries across multiple databases from the SQLite command line though (eg http://stackoverflow.com/questions/6824717/sqlite-how-do-you-join-tables-from-different-databases ), so maybe there is way..? – psychemedia Sep 25 '16 at 12:59

0 Answers0