API requests with PHP slow loading time

Question

I'm using PHP to get price data and volume data from several bitcoin exchanges but when you load the page it takes close to 20 seconds. How can I make the load time better? I think it has something to do with the curl.

 <?php
    function getData($url) {
    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    $rawData = curl_exec($curl);
    curl_close($curl);
    return json_decode($rawData, true);
    }
    //BTC Volume LocalBitcoins
    $BTCVolumeLocal = getData('https://localbitcoins.com/bitcoinaverage/ticker-all-currencies/');
    $LocalVolume = $BTCVolumeLocal["USD"]["volume_btc"];

    //BTC Volume BTCE
    $BTCVolumeBTCE = getData('https://btc-e.com/api/3/ticker/btc_usd');
    $BTCEVolume = $BTCVolumeBTCE["btc_usd"]["vol_cur"];

    //BTC Volume Bitstamp
    $BTCVolumeStamp = getData('https://www.bitstamp.net/api/ticker/');
    $StampVolume = $BTCVolumeStamp["volume"];

    //BTC Volume Bitfinex
    $BTCVolumeFinex = getData('https://api.bitfinex.com/v1/pubticker/btcusd');
    $FinexVolume = $BTCVolumeFinex["volume"];

    //BTC Volume OKCoin
    $BTCVolumeOK = getData('https://www.okcoin.com/api/ticker.do?ok=1');
    $OKCoinVolume = $BTCVolumeOK["ticker"]["vol"];

    //BTC Volume LakeBTC
    $BTCVolumeLake = getData('https://www.lakebtc.com/api_v1/ticker');
    $LakeVolume = $BTCVolumeLake["USD"]["volume"];

    //Totals the Volumes
    $TotalVolume = $LakeVolume + $FinexVolume + $OKCoinVolume + $StampVolume + $BTCEVolume + $LocalVolume;
    //Percents of Total Volume
    $BTCEPercent = $BTCEVolume / $TotalVolume;
    $StampPercent = $StampVolume / $TotalVolume;
    $FinexPercent = $FinexVolume / $TotalVolume;
    $OKPercent = $OKCoinVolume / $TotalVolume;
    $LakePercent = $LakeVolume / $TotalVolume;
    $LocalPercent = $LocalVolume / $TotalVolume;

    //BTC Price BTCE
    $BTCPriceBTCE = getData('https://btc-e.com/api/3/ticker/btc_usd');
    $BTCEPrice = $BTCPriceBTCE["btc_usd"]["last"];

    //BTC Price Bitstamp
    $BTCPriceStamp = getData('https://www.bitstamp.net/api/ticker/');
    $StampPrice = $BTCPriceStamp["last"];

    //BTC Price Bitfinex
    $BTCPriceFinex = getData('https://api.bitfinex.com/v1/pubticker/btcusd');
    $FinexPrice = $BTCPriceFinex["last_price"];

    //BTC Price OKCoin
    $BTCPriceOK = getData('https://www.okcoin.com/api/ticker.do?ok=1');
    $OKPrice = $BTCPriceOK["ticker"]["last"];

    //BTC Price LakeBTC
    $BTCPriceLake = getData('https://www.lakebtc.com/api_v1/ticker');
    $LakePrice = $BTCPriceLake["USD"]["last"];

    //BTC Price LocalBitcoins
    $BTCPriceLocal = getData('https://localbitcoins.com/bitcoinaverage/ticker-all-currencies/');
    $LocalPrice = $BTCPriceLocal["USD"]["avg_1h"];

    //BTC Price * Percent
    $BTCEPricePercent = $BTCEPrice * $BTCEPercent;
    $StampPricePercent = $StampPrice * $StampPercent;
    $FinexPricePercent = $FinexPrice * $FinexPercent;
    $OKPricePercent = $OKPrice * $OKPercent;
    $LakePricePercent = $LakePrice * $LakePercent;
    $LocalPricePercent = $LocalPrice * $LocalPercent;

    //Bitcoin Price
    $bitcoinPrice = round($LakePricePercent + $OKPricePercent + $FinexPricePercent + $StampPricePercent + $BTCEPricePercent + $LocalPricePercent, 2);

    ?>

Use `curl`. Faster. Sexier. MOAR BETTER! Also possible duplicate: http://stackoverflow.com/questions/3629504/php-file-get-contents-very-slow-when-using-full-url — Kisaragi, Oct 15 '15 at 15:09
The reason that this is taking lots of time is because all those requests happen serially. Unfortunately, it appears there is no nice way to make requests asynchronously in PHP. — Phylogenesis, Oct 15 '15 at 15:21
I changed my getData function and added a context variable and it is still pretty slow. — Darkstar, Oct 15 '15 at 15:26
Ok I used curl and it loads in like 5 seconds now. Little better than before but still not very good. Is there anything else I can do for speed? — Darkstar, Oct 15 '15 at 15:33
As I said, the requests are happening serially (one request is made and the response is waited for, then the next request is made, and so on). There is no way in stock PHP to trigger all the requests to happen at once. If you have control of the server, perhaps you can look into installing/using [pthreads](http://php.net/manual/en/book.pthreads.php). But you may just have to use a different language that supports asynchronous requests. — Phylogenesis, Oct 15 '15 at 15:36
If you're making external requests, it's difficult to count on a quick load time since you're also relying on the service to always be fast and available. Have you thought about loading the page and then making those requests with javascript / ajax and showing the user a loading symbol while it's coming up? That may be a better user experience. — Jage, Oct 15 '15 at 15:38
You could add some asynchronous functionality like [Gearman](http://gearman.org). — Jan, Oct 15 '15 at 16:08
I'm thinking the best way is to set a CRON to run every minute and save the data to a file and then just request the data from the file instead of from the exchange sites every time. How would I go about doing this? — Darkstar, Oct 15 '15 at 16:43
Well something might make it a little faster is using only 1 curl handle for all requests `$ch=curl_init()` and after you finish the http request you make `curl_reset($ch);` that will reset the curl handle options to default so you could use the handle again instead of making another 11 new handles and in the end `curl_close($ch)` but as an answer says if the API is slow then you can't do anything about it — Mohamed Belal, Oct 15 '15 at 18:04
Save the curl_init in a global variable( outside of function getData) and in the function getData replace $ch with $GLOBALS['ch'] and curl_init with curl_reset then in the end of the script curl_close($GLOBALS['ch']); check if that makes it little faster — Mohamed Belal, Oct 15 '15 at 18:15
I think you didn't understand what I said :p should I add it as an answer ? — Mohamed Belal, Oct 15 '15 at 18:20

score 5 · Answer 1 · answered Oct 15 '15 at 15:45

If the APIs don't respond fast enough and they are not under your control, you probably won't be able to change this.

Querying them in parallel might speed things up, but as mentioned in the comments, that's usually not that easy using PHP.

If it's just about the loading time of your page, you could cache the result of the API queries:

Put the API requests in a cronjob, that's called automatically every x minutes. Depending on how fast the input data is changing, this can be as low as 1 minute (which unfortunately means you'll create a lot of traffic), or you could query it once an hour if that's okay for your calculation.
Save the result to a local database or cache (MySQL, Redis, Memcache, ...)
For your calculation only read from your local copy of the values, which will be a lot faster than querying the services every time

This seems like the best solution. I'm not very familiar with PHP though so how would I go about saving the results? — Darkstar, Oct 15 '15 at 22:05
That depends on what you have or can use. Do you have a MySQL Database, or can you install any software on the machine (is it a root server)? — Pampy, Oct 16 '15 at 13:13

fico7489 · Answer 2 · 2015-10-15T17:20:23.813

2

You have 12 service calls, and 20 seconds is real. I'm not familiar with your app but you can consider implementing some caching (Save downloaded data to files and later load from downloaded files).

$url = 'http://www.example.com';
$content = '';
$file_name = 'tmp/cache.txt';

if(file_exists($file_name))){
    $content = file_get_contents($file_name);
}else{
    $content = file_get_contents($url);
    //or $content = getData($url);
    file_put_contents($file_name, $content);
}


function getData($url) {
    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    $rawData = curl_exec($curl);
    curl_close($curl);
    return json_decode($rawData, true);
}

edited Oct 15 '15 at 17:20

answered Oct 15 '15 at 15:40

fico7489

7,931
7
55
89

How can I save them to a file? I'm not familiar with PHP – Darkstar Oct 15 '15 at 16:56
Is there any way to do it without file_get_contents. According to some other people that is a lot slower than curl. – Darkstar Oct 15 '15 at 17:17
ofc, look at my answer now. – fico7489 Oct 15 '15 at 17:21

score 2 · Answer 3 · answered Oct 15 '15 at 23:01

I found that using ajax the results all came back reasonably quickly because the requests were running asynchronously and this is, so far, work in progress..... If others here can see the benefit I thought I perceived from this approach no doubt they will nail it.

The idea is a js function sends off a series of ajax requests to the php script that then sends the curl request to the various bitcoin urls given. Each request is setup to carry with it the fields to be returned from the request - the php script then drills down through the data and finds that info and returns to js.

Calculating the various percentages remains an issue as you don't necessarily know when all requests have finished. I guess promises might be useful?

Certainly using ajax allows the page to load quickly - then the results come back 'as and when' until finished.....

bitcoin.php
-----------
<?php
    if( $_SERVER['REQUEST_METHOD']=='POST' ){

        $cacert='c:\wwwroot\cacert.pem';


        /* Two of the three params sent via ajax */
        $url=$_POST['url'];
        $fields=$_POST['fields'];

        /* $fields might be a comma separated list of fields, or simply one - we want an array */
        $params=( !empty( $fields ) && strstr( $fields, ',' ) ) ? explode( ',', $fields ) : (array)$fields;


        $curl=curl_init( $url );

        if( parse_url( $url, PHP_URL_SCHEME )=='https' ){
            curl_setopt( $curl, CURLOPT_SSL_VERIFYPEER, FALSE );
            curl_setopt( $curl, CURLOPT_SSL_VERIFYHOST, 2 );
            curl_setopt( $curl, CURLOPT_CAINFO, realpath( $cacert ) );
        }

        curl_setopt( $curl, CURLOPT_URL, $url );
        curl_setopt( $curl, CURLOPT_HEADER, false );
        curl_setopt( $curl, CURLOPT_FRESH_CONNECT, true );
        curl_setopt( $curl, CURLOPT_FORBID_REUSE, true );
        curl_setopt( $curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1 );
        curl_setopt( $curl, CURLOPT_CLOSEPOLICY, CURLCLOSEPOLICY_OLDEST );
        curl_setopt( $curl, CURLOPT_BINARYTRANSFER, true );
        curl_setopt( $curl, CURLOPT_AUTOREFERER, true );
        curl_setopt( $curl, CURLOPT_CONNECTTIMEOUT, 10 );
        curl_setopt( $curl, CURLOPT_RETURNTRANSFER, true );
        curl_setopt( $curl, CURLOPT_USERAGENT, 'Bitcoin Fuzzer' );


        $data=trim( curl_exec( $curl ) );
        $status=curl_getinfo( $curl,CURLINFO_HTTP_CODE );
        curl_close( $curl );



        /* Deal with the response data */
        if( $status==200 ){

            $json=json_decode( $data );
            /* 
                Hack: 
                there must be a better way to drill down dynamically through
                an object structure but I was too tired to think straight

                This simply echos back the response for javascript to deal with
                but I guess one could make use of sessions to do the calculations
                in php rather than js
            */
            switch( count( $params ) ){
                case 1:
                    $p1=$params[0];
                    print_r( $json->$p1 );
                break;
                case 2:
                    $p1=$params[0];
                    $p2=$params[1];
                    print_r( $json->$p1->$p2 );
                break;
                case 3:
                    $p1=$params[0];
                    $p2=$params[1];
                    $p3=$params[2];
                    print_r( $json->$p1->$p2->$p3 );
                break;
            }
        }
    }
?>

the html page - head:

<script type='text/javascript' charset='utf-8'>
    /* simple ajax function */
    function _ajax( url, options ){
        var req=new XMLHttpRequest();
        var callback=options.hasOwnProperty('callback') ? options.callback : false;
        if( !callback ) return false;


        var headers={
            'Accept': "text/html, application/xml, application/json, text/javascript, "+"*"+"/"+"*"+"; charset=utf-8",
            'Content-type': 'application/x-www-form-urlencoded',
            'X-Requested-With': 'XMLHttpRequest'
        };

        var params=[];
        if( options.hasOwnProperty('params') && typeof( options.params )=='object' ){
            for( var n in options.params ) params.push( n + '=' + options.params[n] );
        }
        var args=options.hasOwnProperty('args') ? options.args : options;

        req.onreadystatechange=function(){
            if( req.readyState==4 ) {
               if( req.status==200 ) options.callback.call( this, req.response, args );
               else console.warn( 'Error: '+req.status+' status code returned' );
            }
        }

        req.open( 'POST', url, true );
        for( header in headers ) req.setRequestHeader( header, headers[ header ] );
        req.send( params.join('&') );
    }

    var volumes=[];
    var prices=[];
    var tot_vols=0;
    var tot_prices=0;

    function bitcoin(){
        var btc={
            vols:{
                'https:\/\/localbitcoins.com\/bitcoinaverage\/ticker-all-currencies\/':['USD','volume_btc'],
                'https:\/\/btc-e.com\/api\/3\/ticker\/btc_usd':['btc_usd','vol_cur'],
                'https:\/\/www.bitstamp.net\/api\/ticker\/':['volume'],
                'https:\/\/api.bitfinex.com\/v1\/pubticker\/btcusd':['volume'],
                'https:\/\/www.okcoin.com\/api\/ticker.do?ok=1':['ticker','vol'],
                'https:\/\/www.lakebtc.com\/api_v1\/ticker':['USD','volume']
            },
            prices:{
                'https:\/\/btc-e.com\/api\/3\/ticker\/btc_usd':['btc_usd','last'],
                'https:\/\/www.bitstamp.net\/api\/ticker\/':['last'],
                'https:\/\/api.bitfinex.com\/v1\/pubticker\/btcusd':['last_price'],
                'https:\/\/www.okcoin.com\/api\/ticker.do?ok=1':['ticker','last'],
                'https:\/\/www.lakebtc.com\/api_v1\/ticker':['USD','last'],
                'https:\/\/localbitcoins.com\/bitcoinaverage\/ticker-all-currencies\/':['USD','avg_1h']
            }
        };
        var url;
        var vols=btc.vols;
        var prices=btc.prices;

        for( url in vols ){
            getbitcoin.call( this, url, vols[url], 'volumes' );
        }

        for( url in prices ){
            getbitcoin.call( this, url, vols[url], 'prices' );  
        }
    }


    function getbitcoin( url, fields, type ){
        var options={
            callback:cbbtc,
            method:'POST',
            params:{
                'fields':fields,
                'type':type,
                'url':url
            }
        };
        _ajax.call( this, '/test/bitcoin.php', options );
    }

    function cbbtc(r,o){
        switch( o.params.type ){
            case 'volumes':
                tot_vols += parseFloat( r );
                volumes.push( parseFloat( r ) );
                document.getElementById('btc_vols').value=tot_vols;
            break;
            case 'prices':
                tot_prices += parseFloat( r );
                prices.push( parseFloat( r ) );
                document.getElementById('btc_prices').value=tot_prices;
            break;  
        }
    }           

    /* launch the function when the page is ready */
    document.addEventListener( 'DOMContentLoaded', bitcoin, false );
</script>

html page - body:

/* These simply receive values for testing */
<input type='text' id='btc_vols' />
<input type='text' id='btc_prices' />

Wouldn't it be easier to just request the API data every 1 minute or so and save it to a file on the server. Then when someone loads the page it is basically instant because the data has already been downloaded and you can request it locally. — Darkstar, Oct 15 '15 at 23:23

score 1 · Answer 4 · edited May 23 '17 at 12:07

1

Multithreading and/or caching, friend.

Multithreading in PHP is no tougher than in any other language. You can find instructions here: How can one use multi threading in PHP applications

You can use simple caching with Memcache or Redis, and there are all kinds of tutorials and helps on the web to use these to cache stuff in PHP.

edited May 23 '17 at 12:07

Community

1
1

answered Oct 15 '15 at 22:59

Rob Bailey

1,747
15
18

API requests with PHP slow loading time

4 Answers4