How do I store mostly static data from a JSON api?

Question

My php project is using the reddit JSON api to grab the title of the current page's submission.

Right now I am doing running some code every time the page is loaded and I'm running in to some problems, even though there is no real API limit.

I would like to store the title of the submission locally somehow. Can you recommend the best way to do this? The site is running on appfog. What would you recommend?

This is my current code:

<?php

/* settings */

$url="http://".$_SERVER['HTTP_HOST'].$_SERVER['REQUEST_URI'];

$reddit_url = 'http://www.reddit.com/api/info.{format}?url='.$url;

$format = 'json'; //use XML if you'd like...JSON FTW!
$title = '';

/* action */
$content = get_url(str_replace('{format}',$format,$reddit_url)); //again, can be xml or json
if($content) {
    if($format == 'json') {
        $json = json_decode($content,true);
        foreach($json['data']['children'] as $child) { // we want all children for this example
            $title= $child['data']['title'];
        }
    }
}

/* output */


/* utility function:  go get it! */
function get_url($url) {
    $ch = curl_init();
    curl_setopt($ch,CURLOPT_URL,$url);
    curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
    curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,1);
    $content = curl_exec($ch);
    curl_close($ch);
    return $content;
}
?>

Thanks!

http://stackoverflow.com/questions/1435072/how-to-generate-json-using-php — jsteinmann, Dec 08 '12 at 05:08

Baba · Accepted Answer · 2012-12-17T09:47:18.897

Introduction

Here is a modified version of your code

$url = "http://stackoverflow.com/";
$loader = new Loader();
$loader->parse($url);
printf("<h4>New List : %d</h4>", count($loader));
printf("<ul>");
foreach ( $loader as $content ) {
    printf("<li>%s</li>", $content['title']);
}
printf("</ul>");

Output

New List : 7

New podcast from Joel Spolsky and Jeff Atwood.
Good site for example code/ Pyhton
stackoverflow.com has clearly the best Web code ever conceived in the history of the Internet and reddit should better start copying it.
A reddit-like, OpenID using website for programmers
Great developer site. Get your questions answered and by someone who knows.
Stack Overflow launched into public
Stack Overflow, a programming Q & A site. & Reddit could learn a lot from their interface!

Simple Demo

The Problem

I see some things you want to achieve here namely

I would like to store the title of the submission locally somehow
Right now I am doing running some code every time the page is loaded

From what i understand you need is a simple cache copy of your data so that you don't have to load the url all the time.

Simple Solution

A simple cache system you can use is memcache ..

Example A

$url = "http://stackoverflow.com/";

// Start cache
$m = new Memcache();
$m->addserver("localhost");
$cache = $m->get(sha1($url));

if ($cache) {
    // Use cache copy
    $loader = $cache;
    printf("<h2>Cache List: %d</h2>", count($loader));
} else {

    // Start a new Loader
    $loader = new Loader();
    $loader->parse($url);
    printf("<h2>New List : %d</h2>", count($loader));
    $m->set(sha1($url), $loader);
}

// Oupput all listing
printf("<ul>");
foreach ( $loader as $content ) {
    printf("<li>%s</li>", $content['title']);
}
printf("</ul>");

Example B

You can use Last Modification Date as the cache key as so that you would only save new copy only if the document is modified

$headers = get_headers(sprintf("http://www.reddit.com/api/info.json?url=%s",$url), true);
$time = strtotime($headers['Date']); // get last modification date 
$cache = $m->get($time);

if ($cache) {
    $loader = $cache;
}

Since your class implements JsonSerializable you can json encode your result and also store in a Database like MongoDB or MySQL

 $data = json_encode($loader);
 // Save to DB

Class Used

class Loader implements IteratorAggregate, Countable, JsonSerializable {
    private $request = "http://www.reddit.com/api/info.json?url=%s";
    private $data = array();
    private $total;

    function parse($url) {
        $content = json_decode($this->getContent(sprintf($this->request, $url)), true);
        $this->data = array_map(function ($v) {
            return $v['data'];
        }, $content['data']['children']);
        $this->total = count($this->data);
    }

    public function getIterator() {
        return new ArrayIterator($this->data);
    }

    public function count() {
        return $this->total;
    }

    public function getType() {
        return $this->type;
    }

    public function jsonSerialize() {
        return $this->data;
    }


    function getContent($url) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 1);
        $content = curl_exec($ch);
        curl_close($ch);
        return $content;
    }
}

score 0 · Answer 2 · answered Dec 10 '12 at 10:50

I'm not sure what your question is exactly but the first thing that pops is the following:

foreach($json['data']['children'] as $child) { // we want all children for this example
    $title= $child['data']['title'];
}

Are you sure you want to overwrite $title? In effect, that will only hold the last $child title.

Now, to your question. I assume you're looking for some kind of mechanism to cache the contents of the requested URL so you don't have to re-issue the request every time, am I right? I don't have any experience with appFog, only with orchestra.io but I believe they have the same restrictions regarding writing to files, as in you can only write to temporary files.

My suggestion would be to cache the (processed) response in either:

APC shared memory with a short TTL
temporary files
database

You could use the hash of the URL + arguments as the lookup key, doing this check inside get_url() would mean you wouldn't need to change any other part of your code and it would only take ~3 LOC.

score 0 · Answer 3 · answered Dec 16 '12 at 10:32

After this:

if($format == 'json') {
    $json = json_decode($content,true);
    foreach($json['data']['children'] as $child) { // we want all children for this example
        $title = $child['data']['title'];
    }
}

}`

Then store in a json file and dump it into your localfolder website path

$storeTitle = array('title'=>$title)

$fp = fopen('../pathToJsonFile/title.json'), 'w'); fwrite($fp, json_encode($storeTitle)); fclose($fp);

Then you can always call the json file next time and decode it and extract the title into a variable for use

score 0 · Answer 4 · answered Dec 17 '12 at 05:23

i usually just store the data as is as a flat file, like so:

<?php
define('TEMP_DIR', 'temp/');
define('TEMP_AGE', 3600);

function getinfo($url) {
    $temp = TEMP_DIR . urlencode($url) . '.json';

    if(!file_exists($temp) OR time() - filemtime($temp) > TEMP_AGE) {
        $info = "http://www.reddit.com/api/info.json?url=$url";
        $json = file_get_contents($info);
        file_put_contents($temp, $json);
        }
    else {
        $json = file_get_contents($temp);
        }

    $json = json_decode($json, true);
    $titles = array();
    foreach($json['data']['children'] as $child) {
        $titles[] = $child['data']['title'];
        }
    return $titles;
    }

$test = getinfo('http://imgur.com/');
print_r($test);

PS. i use file_get_contents to get the json data, you might have your own reasons to use curl. also i don't check for format, cos clearly you prefer json.

How do I store mostly static data from a JSON api?

4 Answers4

Introduction

The Problem

Simple Solution

Class Used