aiohttp.request fails on download with status code 404, but synchronous download succeed

Question

I am having a strange problem with an API request. The problem is that using requests or browser, the url I am using works fine. But using an asynchronous servicer the URL fails and returns a status code 404.

Here is my code: I have build an asynch downloader function get:

""" download functions for asynchronous download """
import asyncio
from pathlib import Path
from typing import Union, Dict, Any

import aiohttp

HTTP_DEFAULT_TIME_OUT = 60 * 10


async def _get(
    url: Union[Path, str],
    timeout: aiohttp.ClientTimeout = aiohttp.ClientTimeout(),
    **kwargs: Union[aiohttp.BasicAuth, Dict[str, Any]]
) -> bytes:
    """
    Base function downloader with aiohttp and async session

    Args:
        url: remote path to file
        timeout: how long should it max take to download the file
        **kwargs: parameters to pass to the aiohttp get function:
            auth: aiohttp.BasicAuth("", "")
            params: Dict[str, Any]
            headers: Dict[str, Any]

    Returns:
        a bytes object
    """
    kwargs["timeout"] = timeout
    async with aiohttp.request("GET", url, **kwargs) as response:
        return await response.read()


def get(
    url: Union[Path, str],
    timeout: int = HTTP_DEFAULT_TIME_OUT,
    **kwargs: Union[aiohttp.BasicAuth, Dict[str, Any]]
) -> bytes:
    """
    download function handling async download with aiohttp

    Args:
        url: remote path to file
        timeout: how long should it max take to download the file
        **kwargs: parameters to pass to the aiohttp get function:
            auth: aiohttp.BasicAuth("", "")
            params: Dict[str, Any]
            headers: Dict[str, Any]

    """
    return asyncio.get_event_loop().run_until_complete(
        _get(url, aiohttp.ClientTimeout(timeout), **kwargs)
    )

remote_file='https://api.eumetsat.int/data/download/collections/EO%3AEUM%3ADAT%3AMSG%3AHRSEVIRI/products/MSG4-SEVI-MSG15-0100-NA-20210916121242.731000000Z-NA?access_token=43e1b232-02ed-30db-9d17-363f00a4a1f0'

Note: token will expire.

Using requests works fine. But using my own get function yields to:

get(remote_file)
b'<am:fault xmlns:am="http://wso2.org/apimanager"><am:code>404</am:code><am:type>Status report</am:type><am:message>Runtime Error</am:message><am:description>No matching resource found for given API Request</am:description></am:fault>'

Downloading other stuff with my get function works fine, too.

So does anyone know why the server returns a 404 when trying to access asynchronously ?

My previous question was closed because some of you thought it is a similar issue to this: But the reason for this issue was an 443 ssl certification error. I am still receiving 404 If I apply an additional Connector the behaviour is the same.

connector=aiohttp.TCPConnector(verify_ssl=False)
get(remote_file, connector=connector)

Using: python==3.9.7 aiohttp==3.7.4

Please let me know If the token expired – dl.meteo Sep 21 '21 at 08:56 — dl.meteo, Sep 21 '21 at 08:56

score 0 · Answer 1 · answered Sep 21 '21 at 13:50

The reason is the URL encoding. For this yarl can be used:

import yarl


remote_file = 'https://api.eumetsat.int/data/download/collections/EO%3AEUM%3ADAT%3AMSG%3AHRSEVIRI/products/MSG4-SEVI-MSG15-0100-NA-20210916121242.731000000Z-NA?access_token=43e1b232-02ed-30db-9d17-363f00a4a1f0'
remote_file_encoded = yarl.URL(remote_file, encoded=True)

aiohttp.request fails on download with status code 404, but synchronous download succeed

1 Answers1