0

The story:

  • I've coded in vba excel for years creating financial models.
  • I've outgrown it and was steered towards python which I am learning.
  • Each day I have a 5.5mb csv that I would like to upload to a database which I have not built yet.
  • It seems MySQL is the way to go unless someone has other suggestions.

The question is this: I have 4 years of data in csv format (5.5gb).

  • Are there limitations on table size for database building?
  • Should I consider another database platform?
  • And can I construct an automation module using python to open a csv file (dumped to my server or email) to then update a table in mysql?
Guillaume Jacquenot
  • 11,217
  • 6
  • 43
  • 49
Silverfoxner333
  • 11
  • 1

1 Answers1

0

There is no limitation to my MySQL when it comes to storage, though you should look at the difference between the community version and the paid version( ie; the enterprise verision).

I think you've made the right choice by choosing python for your data processing. I also work with data from multiple sources each in the gigbit range and, I use python and MySQL. I suggest you look into the following python libraries:

  • SQLalchemy
  • Pandas

Btw, you can definitely open up any csv file with python. Just provide the path and use the csv library (or just use pandas)

And yes you can automate the process of email to server, keep in mind you'll need to use libraries like 'os'.

Furthermore, there is a lot of helpful info out there thanks to python community. For starters, you can use the following link as a refrence:

Import multiple csv files into pandas and concatenate into one DataFrame

In case you also need to import data from an Excel file using python, pandas can also do that(or you could look into the xlrd library).

Community
  • 1
  • 1
Ashraf Khalifa
  • 1