1

I recently written a kafka consumer in my django project which is running in the background and there's a mysql query which reads inside the consumer and the process is very light as I don't have that much of data. But I'm getting (2006, 'MySQL server has gone away)

def job_listener():
    listener = KafkaConsumer(topic, group_id=group_id, bootstrap_servers=settings.KAFKA_BOOTSTRAP_SERVERS, auto_offset_reset=settings.KAFKA_AUTO_OFFSET_RESET, enable_auto_commit=settings.KAFKA_ENABLE_AUTO_COMMIT)

    for message in listener:
        try:
            data = json.loads(message.value)
            logger.info("Data Received {}".format(data))
        except Exception as e:
            data = None
            logger.error("Failed to parse the message with exception {}".format(e))
        if data is not None:
            try:
               job = Job.objects.get(pk=data['id'])
            except ObjectDoesNotExist:
               logger.error("Keyword doesn't exist with data {}".format(data))
                   
                listener.commit({TopicPartition(topic, message.partition): OffsetAndMetadata(message.offset+1, '')})
        except Exception as e:
            logger.error("Failed to read data {} with exception {}".format(data, e))
            listener.seek(TopicPartition(topic,
                                             message.partition), message.offset)
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
D.Sunil
  • 21
  • 6
  • I don't think we can learn much from this code as the code is an interaction between django and kafka, while the error occurs between kafka and mysql. You probably need to check from kafka / mysql logs how kafka translated the python code into mysql requests and which of those requests triggered the gone away error. – Shadow Nov 30 '22 at 11:42
  • 1
    From the logs Job.objects.get(pk=data['id']) this read MySQL operation is causing an issue because there's no connection to MySql but Django doesn't know about the connectivity with MySql it returns an error. – D.Sunil Nov 30 '22 at 11:59
  • 1
    WProbably kafka connects to mysql using an sql query, which causes the error to appear. You need to find that query. – Shadow Nov 30 '22 at 12:38
  • Removed Kafka tags since error is unrelated to Kafka. Please [edit] your question to include the entire traceback and correct the indentation of your code. Also, you're not doing anything with the result of `Job.objects.get`, so do you really need that query? – OneCricketeer Nov 30 '22 at 14:33
  • @Shadow Kafka doesn't connect to mysql on its own. The error here is logged by Python/Django mysql client. I assume that's `Job.objects.get` in the code shown – OneCricketeer Nov 30 '22 at 14:36
  • @OneCricketeer I don't think we have any clue what Job.objects.get does. It can equally connect to either or both of Kafka and/or mysql – Shadow Nov 30 '22 at 14:53
  • @Shadow Well, after `for message in listener`, there is no Kafka interaction in the shown code. And `SomeClass.objects.get` corresponds to [Django model API](https://docs.djangoproject.com/en/4.1/ref/models/instances/#django.db.models.Model) – OneCricketeer Nov 30 '22 at 17:43
  • @OneCricketeer I have just written a read query to test the behavior, it seems like the MySQL connection is not persisting and getting closed after some time. – D.Sunil Dec 01 '22 at 09:22
  • Thank you for your question. I have the same error. I'm search for find this problem and fine this paper : https://tacgomes.com/2020/03/29/the-strange-case-of-mysql-gone-away/ If you find other solution please tell me. – Navid Dec 13 '22 at 12:57
  • Hi @Navid, I figured out the cause and reason is MySQL closes all its idle connections if there's no user activity during the last 8 hours(wait_timeout), but Django doesn't know that the connection is being closed from the MySQL side so it was trying to use the same connection in my case. However, if you found out that it's the same issue then you could catch that exception and do db.close_old_connections(), then it will resolve the issue. But there could be many possibilities why this error could occur, you can go through this link https://dev.mysql.com/doc/refman/8.0/en/gone-away.html . – D.Sunil Dec 13 '22 at 18:04

0 Answers0