13

I'm a new oracle learner. I'm trying to write a pandas dataframe into an oracle table. After I have made research online, I found the code itself is very simple, but I don't know why my code doesn't work.

I have read the pandas dataframe from my local file:

import cx_Oracle
import pandas as pd
import os

dir_path = os.path.dirname(os.path.realpath("__file__"))
df = pd.read_csv(dir_path+"/sample.csv")

Now print df, the dataframe df shold be like this:

   DATE            YEAR     MONTH      SOURCE      DESTINATION
0  11/1/2017 1:00  2017     1          AL          CO  
1  11/2/2017 1:00  2017     5          GA          ID  
2  11/3/2017 1:00  2017     12         GA          MO

Then I create connection with the database by using cx_Oracle, it works. Next I try to write the dataframe df into the table TEST. This table TEST is an empty table which already exist in oracle database, it has columns including DATE, YEAR, MONTH, SOURCE, DESTINATION in oracle. All the datatype matches the df sample data. My code is as follows:

conn_str = u'account/password@host:1521/server'
conn = cx_Oracle.connect(conn_str)

# Write records stored in a DataFrame to a oracle database
df.to_sql('TEST', conn, if_exists='replace') # the error shows here

conn.close()

It shows error:

DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ORA-01036: illegal variable name/number

How to solve the problem? Thank you very much for your time!

MaxU - stand with Ukraine
  • 205,989
  • 36
  • 386
  • 419
Haven Shi
  • 457
  • 5
  • 14
  • 19

3 Answers3

17

I've seen similar questions on SO - it happens when you try to write to Oracle DB using connection object created by cx_Oracle.

Try to create connection using SQL Alchemy:

import cx_Oracle
from sqlalchemy import types, create_engine

conn = create_engine('oracle+cx_oracle://scott:tiger@host:1521/?service_name=hr')

df.to_sql('TEST', conn, if_exists='replace')
MaxU - stand with Ukraine
  • 205,989
  • 36
  • 386
  • 419
  • Thank you for you reply. I tried, then it shows DatabaseError: (cx_Oracle.DatabaseError) ORA-01950: no privileges on tablespace 'xxx_DATA' [SQL: 'INSERT INTO 'TEST' ("index", "DATE", "YEAR", "MONTH",...) VALUES (:"index", :"DATE", :YEAR,...)] – Haven Shi Nov 29 '17 at 13:50
  • @HavenShi, this is a different story. That means you (or your DBA) have to add quota on that tablespace for that oracle user. I think your original question has been answered ;-) – MaxU - stand with Ukraine Nov 29 '17 at 13:55
  • ,can you explain a little more about tablespace? I can write directly into the oracle table TEST, what do I miss? – Haven Shi Nov 29 '17 at 14:09
  • @HavenShi, http://www.dba-oracle.com/t_ora_01950_no+priviledges_on_tablespace_string.htm – MaxU - stand with Ukraine Nov 29 '17 at 14:10
  • I get the error `InvalidRequestError: Could not reflect: requested table(s) not available in Engine` does this mean that despite the confusing documentation `replace: If table exists, drop it, recreate it, and insert data.` I have to create the table, since it won't be created automatically? – Superdooperhero Feb 13 '20 at 15:01
  • Is there any way to automatically create the Oracle table based on the dataframe columns and dtypes? – Superdooperhero Feb 13 '20 at 15:03
  • 1
    Looks like it needed the table name in lowercase and it does not like it if the table exists. Creates everything as CLOBs which is quite annoying. – Superdooperhero Feb 13 '20 at 15:19
  • 1
    @Superdooperhero, you might want to check [this answer](https://stackoverflow.com/a/42769557/5741205) ;) – MaxU - stand with Ukraine Feb 13 '20 at 15:42
2

I am able to load an Oracle table using the following code:

import pandas as pd
import os

creds = {}
creds['tns_admin'] = 'Wallet_Path'
creds['sid'] = 'dev_low'
creds['user'] = 'username'
creds['password'] = pwd

os.environ['TNS_ADMIN'] = creds['tns_admin']


uri = 'oracle+cx_oracle://' + creds['user'] + ':' + creds['password'] + '@' + creds['sid']
df = pd.read_csv("test.csv")
df.to_sql('test', uri, schema='PRD', if_exists='replace')

Instead of connection we need to build and pass an URI.

Note: New Oracle databases (Autonomous) requires a wallet, so we need to set wallet path in TNS_ADMIN environment variable.

Also, I didn't have to import cx_Oracle, I did double check that enter image description here

To ensure, I am not getting fooled, I dropped the table and did commit enter image description here

And I executed above code, it created new table with data.

Hussain Bohra
  • 985
  • 9
  • 15
2

After referring to this solution, I was able to get this done using the following steps.

from sqlalchemy.engine import create_engine

DIALECT = 'oracle'
SQL_DRIVER = 'cx_oracle'
USERNAME = 'your_username' 
PASSWORD = 'your_password'
HOST = 'subdomain.domain.tld' 
PORT = 1521 
SERVICE = 'your_oracle_service_name'
ENGINE_PATH_WIN_AUTH = DIALECT + '+' + SQL_DRIVER + '://' + USERNAME + ':' + PASSWORD +'@' + HOST + ':' + str(PORT) + '/?service_name=' + SERVICE

engine = create_engine(ENGINE_PATH_WIN_AUTH)

After successfully creating the SQLAlchemy engine you can pass it to pandas to_sql() function.

df.to_sql('name_of_sql_table',engine,schema='your_schema')
Ransaka Ravihara
  • 1,786
  • 1
  • 13
  • 30