I’m getting a MemoryError when I try to read a large sql file in Python3.6 with this command:
with open(sql_file, ‘r’) as f:
Is there an alternative to this so I can correctly read my sql file ?
Asked
Active
Viewed 186 times
-2
-
Is it _that_ line that causes the error, or lines inside the `with`block where you actually get the contents of the file? – khelwood Jul 09 '18 at 07:58
-
No it is exactly that line – slidefizz Jul 09 '18 at 07:59
-
1Just opening the file is very unlikely to cause a memory error (a file pointer is only a few bytes), unless you already ate all your memory _before_ this line. – bruno desthuilliers Jul 09 '18 at 08:01
-
Is your sql file a query or a data file? How big is the file and how much RAM is installed on your machine? – DDS Jul 09 '18 at 08:01
-
Also this as absolutely NOTHING to do with sql nor postgres. – bruno desthuilliers Jul 09 '18 at 08:02
-
My sql file is a dump of a database so there is table creation and value insertion in it – slidefizz Jul 09 '18 at 08:03
-
Ok: what format is that dump? (consider adding some lines of your dump to the question) – DDS Jul 09 '18 at 08:04
-
use a memory mapped file in python or read it line by line https://stackoverflow.com/questions/6475328/read-large-text-files-in-python-line-by-line-without-loading-it-in-to-memory – rahul tyagi Jul 09 '18 at 08:06
-
Another option is to import it to RDBMS engine and connect python to the database engine – DDS Jul 09 '18 at 08:08
1 Answers
0
If you have the sql file you should import it into a database and than use a library like MySQLdb for python to make your database requests.
I would guess, that opening sql files directly is a good solution. If you want to work with files, I think csv files are the better solution.
Inuff
- 35
- 1
- 9
-
Not the specific problem, but a better solution at all. It is obvious that large file needs more memory. To AVOID problems, it is a good way to give a person the opportunity to find a better solution. – Inuff Jul 09 '18 at 08:14
-
I need to create the database with python, i’m using psycopg2 to make the link between python and the postgres database. I also get a memoryerror when I execute the sql file with psycopg2 function directly. – slidefizz Jul 09 '18 at 08:17