7

I am trying to understand what is the right way to use aiohttp with Sanic.

From aiohttp documentation, I find the following:

Don’t create a session per request. Most likely you need a session per application which performs all requests altogether. More complex cases may require a session per site, e.g. one for Github and another one for Facebook APIs. Anyway making a session for every request is a very bad idea. A session contains a connection pool inside. Connection reuse and keep-alive (both are on by default) may speed up total performance.

And when I go to Sanic documentation I find an example like this:

This is an example:

from sanic import Sanic
from sanic.response import json

import asyncio
import aiohttp

app = Sanic(__name__)

sem = None

@app.route("/")
async def test(request):
    """
    Download and serve example JSON
    """
    url = "https://api.github.com/repos/channelcat/sanic"

    async with aiohttp.ClientSession() as session:
         async with sem, session.get(url) as response:
         return await response.json()

app.run(host="0.0.0.0", port=8000, workers=2)

Which is not the right way to manage an aiohttp session...

So what is the right way?
Should I init a session in the app and inject the session to all the methods in all layers?

The only issue I found is this but this doesn't help because I need to make my own classes to use the session, and not sanic.
Also found this in Sanic documentation, which says you shouldn't create a session outside of an eventloop.

I am a little confused :( What is the right way to go?

John Moutafis
  • 22,254
  • 11
  • 68
  • 112
Tomer
  • 2,398
  • 1
  • 23
  • 31
  • Hey @Tomer, I was wondering, did you find any of the answers helpful? – John Moutafis Nov 23 '18 at 11:24
  • 2
    @johnMoutafis Thanks! We actually did something similar, with minor changes. First of all the passing loop into ClientSession is deprecated since version 2.0, so we don't do that. Also we don't define a global session using 'global' but we put it on the app. Also don't forget you need to close the ClientSession when the app is closed. – Tomer Nov 25 '18 at 08:46
  • @johnMoutafis, if you agree with me, would love if you can change your answer so I can click on "accept answer" :) – Tomer Nov 25 '18 at 08:47
  • I did some research because what you told me was very interesting, and I updated my answer :D – John Moutafis Nov 26 '18 at 10:00
  • @johnMoutafis Thanks:) Also update the description. (you are no longer use global) – Tomer Nov 27 '18 at 12:23
  • @johnMoutafis BTW, fun fact, we started using aiohttp implementation instead of Sanic:) – Tomer Nov 27 '18 at 12:24
  • Sounds interesting :) (fixed the description as well) – John Moutafis Nov 27 '18 at 12:49

2 Answers2

13

In order to use a single aiohttp.ClientSession we need to instantiate the session only once and use that specific instance in the rest of the application.

To achieve this we can use a before_server_start listener which will allow us to create the instance before the app serves the first byte.

from sanic import Sanic 
from sanic.response import json

import aiohttp

app = Sanic(__name__)

@app.listener('before_server_start')
def init(app, loop):
    app.aiohttp_session = aiohttp.ClientSession(loop=loop)

@app.listener('after_server_stop')
def finish(app, loop):
    loop.run_until_complete(app.aiohttp_session.close())
    loop.close()

@app.route("/")
async def test(request):
    """
    Download and serve example JSON
    """
    url = "https://api.github.com/repos/channelcat/sanic"

    async with app.aiohttp_session.get(url) as response:
        return await response.json()


app.run(host="0.0.0.0", port=8000, workers=2)

Breakdown of the code:

  • We are creating an aiohttp.ClientSession, passing as argument the loop that Sanic apps create at the start, avoiding this pitfall in the process.
  • We store that session in the Sanic app.
  • Finally, we are using this session to make our requests.
adam shamsudeen
  • 1,752
  • 15
  • 14
John Moutafis
  • 22,254
  • 11
  • 68
  • 112
  • Having this setup ended up crashing some of my client requests. Sometimes if you have two coroutines running, first one __aenter__s and awaits a response. Then, second one __aenter__s but finishes its request and __aexit__s the open session before the first one is done. Thus giving me "Session is closed" errors. I'm not 100% sure how ClientSession's async ctx manager is implemented or specifically how ClientSession.exit is implemented, but instantiating a new client session with each request solved my problem. I think it'd be nice if aiohttp team had any opinion regarding this issue. – Charming Robot Mar 28 '19 at 15:08
  • How to propagate app object to another files / modules? Let's say I have multiple files with set of "@app.route". I created app in main and what to do next? – likern Apr 11 '19 at 19:09
  • @likern The way the `init` is implemented, means that the `aiohttp_session` is stored inside the `app` and you can use it from there. – John Moutafis Apr 11 '19 at 19:21
  • @JohnMoutafis I understand that. But if I have another (similar) file with routes, how to get app object itself to this file? – likern Apr 11 '19 at 19:24
  • @JohnMoutafis In all examples and in documentation I see only usage of one file. But I don't understand how to use multiple files with Sanic, when you have a lot of routes. – likern Apr 11 '19 at 19:26
  • @likern Usually (as with Flask) blueprints is the way to go: https://sanic.readthedocs.io/en/latest/sanic/blueprints.html – John Moutafis Apr 11 '19 at 19:35
  • @JohnMoutafis it would be great, if you could extend you example to case with multiple files - how to combine all of that to one goal - initialize ClientSession object once so it would be available in every file - in app object or in blueprints or something else. – likern Apr 11 '19 at 19:45
  • @likern That would bring the answer out of the scope of the question. You can make a new question here on SO with a good minimal example and ask about multiple files etc. – John Moutafis Apr 11 '19 at 22:39
  • I too would like to know how to import this session to blueprints. Otherwise if you try to import the session into a blueprint it crashes the Sanic app with a circular import – xristian Nov 27 '19 at 20:03
3

That is essentially what I am doing.

I created a module (interactions.py) that has, for example a function like this:

async def get(url, headers=None, **kwargs):
    async with aiohttp.ClientSession() as session:
        log.debug(f'Fetching {url}')
        async with session.get(url, headers=headers, ssl=ssl) as response:
            try:
                return await response.json()
            except Exception as e:
                log.error(f'Unable to complete interaction: {e}')
                return await response.text()

Then I just await on that:

results = await interactions.get(url)

I am not sure why that is not the "right way". The session (at least for my needs) can be closed as soon as my request is done.

Adam Hopkins
  • 6,837
  • 6
  • 32
  • 52
  • 2
    Thanks for your answer:) It's not the right way because this is what they say in their documentation. They say you don't need to open a client session (which has a connection pool) every time you need to do a request, but only one time. – Tomer Aug 02 '18 at 14:00
  • Pretty much what I have done with a slight difference. Once change I made to that kinda setup is returning the response and response.json() as a tuple. So response, response_json = await req.get(). That allows you to check response status, headers, etc. I also added timeout and async with timeout. – Christo Goosen Nov 08 '18 at 06:22