Getting more than 100 records with Airtable

Question

I'm building an App that gets a database on Airtable with the Requests library and transform it into a list.

My issue is that I can only get 100 records with the URL solicitation. I tried to find a solution in the Airtable API, but the changes in the URL to insert the maxRecords didn't work.

I'm using this URL to get the data. But it returns just 100 records.

    https://api.airtable.com/v0/appP6G7OJmLzUCoUt/Table%201?api_key=MY_KEY

I've read about maxRecords and pagination, but I couldn't find a way of changing the URL to use that.

Does anyone can help me?

Answer 1

TLDR:
You could try using a recursive function that executes when an offset exists in the http response (json).

Acknowledgment:
This solution took 7 hours of research, troubleshooting, and advice from the great, and powerful Doug. The project uses Alamofire to perform http requests, and SwiftyJson to access the JSON.

Cause:
In the Airtable documentation, they state their rate limit is 100 items per request. If a request has more than 100 items, the request will include an offset.

They give an instruction to include the offset in your next request to get the next 100 results. But, they don't explain or demonstrate how to do it.

Solution:
It's been tested to retrieve 2,565 items from 25 http requests. Written in Swift, the logic is simple:

Write a recursive function with an argument for an optional offset. Call the function without the offset. Check the http response (json) for the offset. If the offset exists, store the http request (json) in an array outside the function. Then, call that same function from inside itself - this time with the offset.

Extended code here.

func requestAirtableRecords(forTable table: String, withTableView tableView: String, withOffset offset: String?, completion: @escaping ([JSON]) -> ()) {
    let parameters: [String: Any] = offset != nil ? ["view": tableView, "offset": offset!] : ["view": tableView]
    do {
        let url: URLRequest = try self.requestRecordsURL(table: table, method: HttpRequest.get, parameters: parameters)!
        Alamofire.request(url).responseJSON { (response) in
            switch response.result {
            case .success(_):
                let json = JSON(response.result.value!)
                self.jsonArray.append(json)
                let nextOffset = json["offset"]
                if nextOffset.exists() {
                    self.requestAirtableRecords(forTable: table, withTableView: tableView, withOffset: nextOffset.stringValue, completion: { _ in
                        completion(self.jsonArray)
                    })
                } else {
                    completion(self.jsonArray)
                }
            case .failure(let error):
                print(error)
            }
        }
    } catch {
        print("Error: Unable to request records from Airtable.")
    }
}

Answer 2

As I saw in other posts, many people were dealing with the same issue. I tried to find solucions, but I coudn't fix it with the URL.

Although, thank God, it is easier to get all data from Airtable in Python with the Airtable API library. (http://airtable-python-wrapper.readthedocs.io/en/master/)

There is a function called get_all(), wich acept the maxRecords argument. Just call x.get_all(), with no argument inside, and the API will return every record in the table.

Answer 3

This is the code I used in Python to solve this issue - in line with the answer by Christian, I used a recursive call. However, I'm posting this code b/c the OP requested an answer in Python.

def get_airtable_data(email_dict=None, offset=None):
        """
        queries airtable and creates a dict of {name: email} from records that are 
        listed as subscribed
        """

    if email_dict is None:
        email_dict = {}
    r = requests.get(request_url, headers={"Authorization": bearer_token}, params={"pageSize": 100, "offset": offset})
    r_json = json.loads(r.text)
    for item in r_json["records"]:
        if item["fields"]["subscribed"] == 1:
            email_dict[item["fields"]["Name"]] = item["fields"]["Primary Email"]

    if r_json["offset"] is None:
        print("all data loaded")
    else:
        try:
            get_airtable_data(email_dict, r_json["offset"])
        except KeyError:
            print("all data loaded")
    return email_dict