How to read data from REST API and insert data in to MongoDB using Python?

Question

How to read data from the REST API and insert that fetched data into MongoDB? I know that we can read REST API data using requests library and obtain the output. But how to insert the obtained output into MongoDB using python?

import requests

url='http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/tas/year/CAN.csv'

response = requests.get(url)

if response.status_code != 200:
    print('Failed to get data:', response.status_code)
else:
    print('First 100 characters of data are')
    print(response.text[:100])

when I run this code I get the below output:

First 100 characters of data are
year,data
1901,-7.67241907119751
1902,-7.862711429595947
1903,-7.910782814025879
1904,-8.15572929382

Instead of printing, I want to insert in to mongodb. for this should I have to store the output in a file and then insert it to mongodb? if so, how to do?

your question is too broad, please, be more specific, add your code snippet with actual REST API you are struggling with — Azat Ibrakov, Sep 03 '18 at 12:32
You can use ``requests`` module to read a REST API and write to MongoDB with ``pymongo``. — ramazan polat, Sep 03 '18 at 12:40

Vijay Anand Pandian · Accepted Answer · 2018-09-15T09:07:15.770

Hope this helps, Let me know if you have any issues,

# *-* coding: utf-8 *-*

import requests

try:
    from pymongo import MongoClient
except ImportError:
    raise ImportError('PyMongo is not installed')


class MongoDB(object):
    def __init__(self, host='localhost', port=27017, database_name=None, collection_name=None):
        try:
            self._connection = MongoClient(host=host, port=port, maxPoolSize=200)
        except Exception as error:
            raise Exception(error)
        self._database = None
        self._collection = None
        if database_name:
            self._database = self._connection[database_name]
        if collection_name:
            self._collection = self._database[collection_name]

    def insert(self, post):
        # add/append/new single record
        post_id = self._collection.insert_one(post).inserted_id
        return post_id


url = 'http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/tas/year/CAN.csv'
response = requests.get(url)
data = response.text
if response.status_code != 200:
    print('Failed to get data:', response.status_code)
else:
    print('First 100 characters of data are')
    print(data[:100])

print('[*] Parsing response text')
data = data.split('\n')
data_list = list()
for value in data:
    if 'year,data' not in value:
        if value:
            value = value.split(',')
            data_list.append({'year': int(value[0]), 'data': float(value[1])})

print(data_list)

print('[*] Pushing data to MongoDB ')
mongo_db = MongoDB(database_name='Climate_DB', collection_name='climate_data')

for collection in data_list:
    print('[!] Inserting - ', collection)
    mongo_db.insert(collection)

Output -

First 100 characters of data are
year,data
1901,-7.67241907119751
1902,-7.862711429595947
1903,-7.910782814025879
1904,-8.15572929382
[*] Parsing response text
[{'year': 1901, 'data': -7.67241907119751}, {'year': 1902, 'data': -7.862711429595947}, {'year': 1903, 'data': -7.910782814025879}, {'year': 1904, 'data': -8.155729293823242}, {'year': 1905, 'data': -7.547311305999756}, {'year': 1906, 'data': -7.684103488922119}, {'year': 1907, 'data': -8.413553237915039}, {'year': 1908, 'data': -7.790929317474365}, {'year': 1909, 'data': -8.23930549621582}, {'year': 1910, 'data': -7.774611473083496}, {'year': 1911, 'data': -8.114446640014648}, {'year': 1912, 'data': -7.885402679443359}, {'year': 1913, 'data': -7.987940311431885}, {'year': 1914, 'data': -7.965937614440918}]
[*] Pushing data to MongoDB 
[!] Inserting -  {'year': 1901, 'data': -7.67241907119751}
[!] Inserting -  {'year': 1902, 'data': -7.862711429595947}
[!] Inserting -  {'year': 1903, 'data': -7.910782814025879}
[!] Inserting -  {'year': 1904, 'data': -8.155729293823242}
[!] Inserting -  {'year': 1905, 'data': -7.547311305999756}
[!] Inserting -  {'year': 1906, 'data': -7.684103488922119}

Install requirements,

pip install pymongo

MongoDB windows setup - https://gist.github.com/vijayanandrp/ddf72c2089015ee3f13be2b1a06bfd08

score 1 · Answer 2 · answered Sep 03 '18 at 12:45

1

You can grab the response as json through requests.

Tutorial: HTTP requests and JSON parsing in Python

Then it's just a matter of inserting the stuff you want as a document into mondoDB. You'll need a client for that; pymongo is a pretty good bet.

Tutorial: http://api.mongodb.com/python/current/tutorial.html

answered Sep 03 '18 at 12:45

nz_21

6,140
7
34
80

1

thank you!! but i dint get how to insert it to the mongodb. Could u please share some code snippets. i have added the code snippet above. @nz_21 – swetha reddy Sep 15 '18 at 08:27

How to read data from REST API and insert data in to MongoDB using Python?

2 Answers2