18

My technology stack is Redis as a channels backend, Postgresql as a database, Daphne as an ASGI server, Nginx in front of a whole application. Everything is deployed using Docker Swarm, with only Redis and Database outside. I have about 20 virtual hosts, with 20 interface servers, 40 http workers and 20 websocket workers. Load balancing is done using Ingress overlay Docker network.

The problem is, sometimes very weird things happen regarding performance. Most of requests are handled in under 400ms, but sometimes request can take up to 2-3s, even during very small load. Profiling workers with Django Debug Toolbar or middleware-based profilers shows nothing (timing 0.01s or so)

My question: is there any good method of profiling a whole request path with django-channels? I would like how much time each phase takes, i.e when request was processed by Daphne, when worker started processing, when it finished, when interface server sent response to the client. Currently, I have no idea how to solve this.

Valar
  • 1,833
  • 1
  • 16
  • 18
  • 2
    In case anyone is curious, it appears the author of this post went and asked the Google Group: https://groups.google.com/forum/#!topic/django-users/_aG8Py_r2QY – raiderrobert May 22 '17 at 18:03
  • 1
    Yes, it's my question. I've found some useful methods, one of them is redis monitoring. I'll post detailed description here soon. – Valar May 23 '17 at 16:35
  • Looking forward to hearing what worked for you. – jaywhy13 Aug 02 '17 at 23:57

2 Answers2

1

Django-silk might be helpful to you in profiling the request and database searching time with following reasons:

  1. It is easy to set by simply adding the configs on settings.py of your Django project.
  2. Can be customised: by using the provided decorator, you can profile the function or methods and get their running performance.
  3. Dynamic setting: you can choose to dynamically allocate silk to methods and also set the profiling rate you want during the running time.

As the documentation states:

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection

Note: silk may double your database searching time, so it may cause some trouble if you set it on your production environment. However, the increase from silk will be shown separately on the dash board.

https://github.com/jazzband/django-silk

Lord Vice
  • 11
  • 2
  • While this may answer the question, providing additional context regarding how and/or why it solves the problem would improve the answer's long-term value. – Tiago Martins Peres Oct 17 '18 at 14:16
0

Why not stick a monitoring tool something like Kibana or New Relic and monitor why and what's taking so long for a small payload response. It can tell you the time spent on Python, PostgreSQL and Memcache (Redis).

IVI
  • 1,893
  • 1
  • 16
  • 19