While loop to make API calls until a condition is met

Question

I want to make API calls until a condition is met. I figured I might use a while loop.

I have a JSON response from the server that is paginated.

{
    "services": [
        {
            "id": "ABC12",
            "name": "Networks",
            "description": null,
            "status": "active",
            "teams": [
                {
                    "id": "XYZ12",
                    "type": "team_reference",
                    "summary": "Network Systems ",
                }
            ],
            "acknowledgement_timeout": null,
            "auto_resolve_timeout": null,
            "alert_grouping": "intelligent",
            "alert_grouping_timeout": null,
            "integrations": [],
            "response_play": null,
            "type": "service",
            "summary": "All Events",
        }
     ],
     "limit": 25,
     "offset": 0,
     "total": null,
     "more": true
}

limit - max I can set is 100.

offset - If specified, shows results from that point.

more - If TRUE, there are more results. If FALSE, that is the end.

for more info on this pagination - https://v2.developer.pagerduty.com/docs/pagination

I need to match the name "Networks" and get its corresponding id "ABC12". The problem is, I have to paginate make multiple calls to the API.

I have written this so far.

import requests
import json
import urllib3

# Supress SSL warnings
urllib3.disable_warnings()

# API key
API_KEY = '12345asdfg'

def list_services():

    x = 25
    y = 0
    results = []

    url = f'https://api.pagerduty.com/services/?limit={x}&offset={y}'
    headers = {
        'Accept': 'application/vnd.pagerduty+json;version=2',
        'Authorization': 'Token token={token}'.format(token=API_KEY)
    }
    current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
    results.append(current_page)

    while current_page['more'] == 'True':
        y = y + 1
        current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
        results.append(current_page)
        print(results) # Does not print anything
    print(results) # Prints only the first call results, the while loop 
                   # doesn't seem to work.

if __name__ == '__main__':
    list_services()

the print(results) outside the while loop prints only the first API call results. The while loop doesn't seem to work. But the code compiles without any errors.

1. how do I set the value of x to 25 and make API calls and append the results to results until more is false?

OR

2. how do I make multiple API calls until I find the match. If I found a match, then stop making the call.

Or is there a better cleaner way to do this?

Answer 1

This does not work because you never actually reassign the url variable once y is changed. Also you are checking against 'True' which is a string, not a boolean value. In addition I believe the offset should increase by the amount of results everytime; not just one. For example if on your first call you get results 1-25. Then if you increase y by one, the second call will yield 2-26. Instead you should increase it by the limit. This way on the second call you get results 25-50. Here is how I would do this:

def list_services():

    x = 25
    y = 0
    results = []
    serv_id = None
    flag = False

    url = f'https://api.pagerduty.com/services/?limit={x}&offset={y}'
    headers = {
        'Accept': 'application/vnd.pagerduty+json;version=2',
        'Authorization': 'Token token={token}'.format(token=API_KEY)
    }
    current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
    results.append(current_page)

    for serv_set in current_page['services']:
            if serv_set['name'] == 'Networks':
                serv_id = serv_set['id']
                flag = True

    while current_page['more'] == True and not flag:
        for serv_set in current_page['services']:
            if serv_set['name'] == 'Networks':
                serv_id = serv_set['id']
                break
        y += x
        url = f'https://api.pagerduty.com/services/?limit={x}&offset={y}'
        current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
        results.append(current_page)
        print(results) 
    print(results, serv_id) 

You could further clean this up to avoid some redundancy but this should work. You should also check the status of the API call to ensure that you have a valid response.

Edit:

I edited in the issue dealing with obtaining the id attribute when the name == 'Networks'. Once again you could reduce the redundancy in this a lot but this will get you on the right track. Now serv_id = the id of the service with the name of Networks. If no match is found at the end of the iterations then serv_id will be None.

Answer 2

while current_page['more'] == 'True':

You are checking for a string called 'True' instead of a boolean of True, as is defined in your json file. This could be why your while loop is never executing, and you are not receiving your print statement.

Also, generally for API calls that have more than 1 page of data, you need to specify which page you are getting. Which means you need to reinitialize your payload in your while loop.

For example, if an API has a parameter called "page" that you can pass in, in your while loop you would have to pass in page = 1, page = 2, etc. as a payload.