0

So our python program is running into these errors _mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away'). The problem is that the program accesses the db, hen does a lot of crawling to come back with the results after the mysql connection timeout ... and then its too late.

By logic there are 2 solutions

  • increase mysql connection timeout but this is no option
  • have python check for an open connection, and if closed then re-open it

Some solutions have been found and sound clear possible solution and here about closing and opening cursors.

However we are using models from django .. and I dont know where to implement logic for a check for a connection and reconnect of the connectin is lost check.

question: Where and how can I implement the described logic to re-connect to a lost db connection when using models? (is there some kind of INIT or CONNECT event to access)

Sample code

from django.db import models

class Domain(models.Model):
    name = models.CharField(max_length=100)
    domain = models.CharField(max_length=100, blank=True, null=True)
snh_nl
  • 2,877
  • 6
  • 32
  • 62
  • How are you doing the crawling? Are you running it as part of a view and waiting for that view to return, or otherwise backgrounding the task using a message queue or possibly issuing an RPC to something like scrapyd? – Jon Clements Mar 02 '20 at 18:04

2 Answers2

0

If you close your connection, django will automatically re-open it if it needs to query the database.

So if you know that the problem comes in some function, wether a view or not, you can close the connection right before or at the beginning of that function:

from django.db import connection 
...
connection.close()
rolf82
  • 1,531
  • 1
  • 5
  • 15
0

This is a bug that wont be fixed as you can see here in the Django docs for the workaround you just have to close the connection if your program is going to be idle for a long time.

from django.db import connection
# DO SOMETHING TO THE DATABASE
connection.close()
Eden
  • 76
  • 3
  • HOw would we add this if using db models? for example we would define a model and call it in the code using `brand = Brand.objects.get(code=spider.brand_code)` – snh_nl Mar 25 '20 at 16:15