Trying to get date specific commit history with curl into raw json text file. Issues with Since, Until, and Pagination?

Question

So I am running this in my terminal:

curl https://api.github.com/repos/d3/d3/commits?since=2016-07-23T00:00:00Z&until=2017-07-23T23:59:59Z

and I think I am getting what I want, however the result is not complete. I think I am just getting the first page. My next move was to back it up to the earliest entry I was able to pull with:

curl https://api.github.com/repos/d3/d3/commits?since=2016-07-23T00:00:00Z&until=2017-01-26T21:07:19Z

I thought this would give me a separate set of results (and manually concatenate) but its exactly the same as the first line. Adding ?page=1,2,3... did not change my results either.

Also: I tried directing this request into a text file but it was blank when I opened and the command ran in the terminal instead.

curl (url) > YearCommits.txt

What am I doing wrong?

Answer 1

Add quotes around your url otherwise, your shell will interpret the first & and will make the command on the left of & run in background (thus ignoring all the parameters you have added on the right of it) :

curl "https://api.github.com/repos/d3/d3/commits?since=2016-07-23T00:00:00Z&until=2017-07-23T23:59:59Z" 

Refer to this link to traverse pagination.

The following request will retrieve page 2 :

curl "https://api.github.com/repos/d3/d3/commits?since=2016-07-23T00:00:00Z&until=2017-07-23T23:59:59Z&page=2" 

Here is a bash script that use curl and jq JSON parser to concatenate the JSON results of each request to a file commit_list.json :

#!/bin/bash

repo=d3/d3
output_file=commit_list.json

loop=0
index=1
tmp_file=tmpfile.txt
per_page=100
access_token=12345666799897950400303332323

rm -f $tmp_file
echo "[]" > $output_file

while [ "$loop" -ne 1 ]
do
    data=$(curl -s "https://api.github.com/repos/$repo/commits?access_token=$access_token&since=2014-07-23T00:00:00Z&until=2017-07-23T23:59:59Z&page=$index&per_page=$per_page")

    check_error=$(echo "$data"  | jq 'type!="array" or length == 0')

    if [ "$check_error" == "true" ]; then
        exit 1
    fi

    if [ "$data" == "[]" ]; then
        loop=1
    else
        echo "$data" > $tmp_file
        concat=$(jq -s add $tmp_file $output_file)
        echo "$concat" > $output_file
        size=$(jq '. | length' $output_file)
        echo "computed $index page - fetched total commit array size of : $size"
        index=$((index+1))
    fi
done