1

I have found topics on ways to merge JSON data but nothing specifically like what I'm trying to do. The API I'm requesting data from has an upper limit of 1000 results at a time from their database. I am able to pull 1000 results just fine if I search between a span of today until one year out. I'm not getting all the results though. Talking to their tech, they suggest that I tailor multiple API requests to pull 1000 from every 30 days in succession, then merge them. I'm not sure how to do that.

The request:

// Get date + 30 days to use in the request:
$endDate = date('Y-m-d',strtotime(date("Y-m-d", mktime()) . " + 30 day"));

$url = PROTOCOL . '://'. ENDPOINT . '?api_key='. API_KEY .'&api_secret='. API_SECRET;
$url .= '&aflt_token=K4RrVSQJFt51F2kyFXDZDFlE9igOtuM4';
$url .= '&format=json';
$url .= '&events=T';
$url .= '&race_headings=F';
$url .= '&race_links=F';
$url .= '&include_waiver=F';
$url .= '&page=1';
$url .= '&results_per_page=1000';
$url .= '&start_date=tomorrow';
$url .= '&end_date='.$endDate;
$url .= '&only_partner_races=F';
$url .= '&only_races_with_results=F';
$url .= '&distance_units=K';

$json = file_get_contents($url);
$json_output = json_decode($json);

They suggest I make similar calls for each successive 30-day period, modifying:

$url .= '&start_date=tomorrow';
$url .= '&end_date='.$endDate;

to the next 30 day span each time. So realistically I could make 6 separate calls to get a full 6 months. The question is how to make these calls then merge the data that's returned. It's all exactly the same/format and all getting put into the same database. Thanks for any insight.

UPDATE: I am able to merge separate, 30-day requests for the max results (1000 apparently) from the API by using 

$merged_data = array_merge(array($json_output, $json_output2, $json_output3, $json_output4, $json_output5));

BUT: the maximum number of requests I can make seems to be limited (only 4, 30-day periods of 1000 results) before I get a memory error: " Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 83 bytes) in /home/runraceitadmin/public_html/dev/runsignup.php on line 165"

So it seems the max number of records I can get from the API this way is 120 days worth. I am looking to pull a full calendar year.

jbone
  • 21
  • 4
  • http://stackoverflow.com/questions/2648968/what-is-a-recursive-function-in-php – Lost F. Feb 09 '15 at 14:16
  • what about the `page` parameter,? The parameter `results_per_page`indicates that the api offers paging. So no need for 1000 subsequent requests, just iterate over the resulting page list with 1000 entries each? – Axel Amthor Feb 09 '15 at 14:23
  • You may look here: [RunSignUp API](https://github.com/RunSignUp-Team/OpenSource/tree/master/RunSignUpApiExamples) – Axel Amthor Feb 09 '15 at 14:29
  • Have you ever thought about simply parse the json and push to database each time? By pushing them to database without deleting old records, you would be effectively merging them. – Sheepy Feb 09 '15 at 14:31
  • Axel -- I thought that also. They are looking for a 1-1000 as the documentation reads and confirmed with their techs. So that's for if you want to offer 25, 50, 100 results paginated. Still the total number that can be pulled per request is limited to 1000. – jbone Feb 09 '15 at 16:22
  • Sheepy -- that did cross my mind (and how I arrived at this post). I'm running Cron jobs now that delete the DB contents, make the API request, then populate the DB with fresh up to date data. I still have to make multiple calls apparently (sounds nuts) to get up to 1000 results for 0-30 days, another 1000 results for 31-60 days, etc. – jbone Feb 09 '15 at 16:25
  • I found some success in making separate API calls, each covering a 30-day successive period and requesting maximum records available (1000 each). I merged the results using array_merge(); HOWEVER, memory won't allow anything past 4, 30-day requests like this. – jbone Feb 10 '15 at 14:33
  • @jbone what if there are more than 1000 results for a 30 day period? and it seems there is too much data to process in memory ... so using some file / database storage would be the option for you. – Guanxi Feb 10 '15 at 14:51
  • @Guanxi That would be the other option I think: setting up individual jobs that make a 30-day request, then populate the common database with just those results, move to the next, etc... It seems really cumbersome (not elegant) but I believe that it would "work" in principle. – jbone Feb 10 '15 at 15:06
  • @jbone is this a one time pull or you need to do that every now and then? Coz if you are going to do that every now and then, I think keeping a local copy would be best and then in future, just keep on updating the change every few days. That way you dont have to do this heavy pull every now and then. – Guanxi Feb 10 '15 at 15:22
  • @Guanxi: Currently it's pulling according to a Cron job that runs every hour (to capture all new events created) since they have an ever-changing database (users add their own new events, etc.). You may be correct. – jbone Feb 10 '15 at 15:25

0 Answers0