I have the following problem. I am getting some data from an api in form of a json.
I want to load this json data into an existing table.
I get the data in following format:
{
"abcd": "blablabla",
"value": [
{
"a": "1",
"b": "2",
"c": "3"
},
{
"a": "4",
"b": "5",
"c": "6"
}
]
}
I am only interested in the data in "value" with "a", "b" and "c" being my columns in the table.
I get only the interesting values by creating a pandas Dataframe:
contents = pandas.json_normalize(response.json(), record_path=['value'])
I can insert this data into mssql with the to_sql method from pandas, however, I would like to do an insert if it doesn't exist or update if a certain key already exists. (e.g. in this example column "a" could be the key column).
I have read about certain sqlalchemy methods with merge and so on, however I was no able to reproduce it in my case.
Here is how I loaded the data into my database:
with open('db_con.json') as db:
con = json.load(db)
con_url = URL.create(
"mssql+pyodbc",
username = con['username'],
password = con['password'],
host = con['host'],
port = con['port'],
database = con['database'],
query = con['query'],
)
engine = sqlalchemy.create_engine(con_url, fast_executemany = False, echo = True)
contents.to_sql(
test_db,
engine,
if_exists='append',
index = False,
chunksize = 2500,
schema = 'test'
)
What would I need to do, to implement such an upsert method?
I guess I cannot use to_sql anymore.