How to multiprocessing Api post request

Question

I successfully post each record one by one from a csv file. However, I'm trying to implement multiprocessing to make it more efficient to handle large data file in the future.

ENDPOINT_URL = 'https://example.com'
headers = {'Api-key': '123abc'}

with open("student.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [dict(d) for d in r ]
    groups = {}

    for k, g in groupby(data, lambda r: (r['name'])):
        #My data mapping

        #for loop to post each record
        post_api = requests.post(ENDPOINT_URL, json=groups, headers=headers)

Is there any new easy way to do the multiprocessing for api request?

Update: I trying to use the grequest but the data i post is null

rs = (grequests.post(u,json=groups, headers=headers) for u in ENDPOINT_URL)
    grequests.map(rs)

    print(grequests.map(rs))

Ramy M. Mousa · Accepted Answer · 2019-04-02T23:39:01.060

0

You can use threading library.

ENDPOINT_URL = 'https://example.com'
headers = {'Api-key': '123abc'}

import threading
with open("student.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [dict(d) for d in r ]
    groups = {}

    for k, g in groupby(data, lambda r: (r['name'])):
        #My data mapping
        t = threading.Thread(target=requests.post,
                             args=(ENDPOINT_URL, groups, headers=headers))
        t.setDaemon(True)
        t.start()

Notice: setDaemon(True) will make the thread automatically kill itself after finishing the job.

Update: if you want to pass keyword arguments check here: keyword arguments threading

edited Apr 02 '19 at 23:39

answered Apr 02 '19 at 23:18

Ramy M. Mousa

5,727
3
34
45

May I know why have to add `args=`?Because it is a syntax error at `json=grops`. – kino.jom Apr 02 '19 at 23:28
I removed the `args` but got this `TypeError: 'Response' object is not callable` – kino.jom Apr 02 '19 at 23:29
I updated my answer, `args` takes the parameter tuple needed by your function. in this case `requests.post`, if you want to provide keyword such as `json=groups` do it via `kwargs` – Ramy M. Mousa Apr 02 '19 at 23:40
May I know how could I print the output? It doesn't support like `print(t.text)`. It is unclear if I `print(t)` – kino.jom Apr 03 '19 at 02:56

How to multiprocessing Api post request

1 Answers1