PHP Search multidimensional array and return key - super super large varying array

Question

Hi there I'm currently using an API built to scrape prices out of an MMO and display them in an array. At the moment this is my code:

    <?php 
ini_set('max_execution_time', 0);
$data = json_decode(file_get_contents("http://www.gw2spidy.com/api/v0.9/json/all-items/all"), true);
foreach($data as $item) {
    echo '<pre>';
    print_r($data);
    echo '</pre>';
}
?>

This returns a MASSIVE array. I saved the html file and it's around 35 megabytes. It has every single price of every single item etc... Okay so basically what I need is to search for an item such as 'Dusk' or by it's itemID '29185'. The prices and array keys change every hour so I've been trying to figure out a dynamic way of extracting this data every time but I'm stumped...

The arrays are structured like:

Array
(
    [count] => 19959
    [results] => Array
        (
    [15875] => Array
        (
            [data_id] => 29185
            [name] => Dusk
            [rarity] => 5
            [restriction_level] => 80
            [img] => https://dfach8bufmqqv.cloudfront.net/gw2/img/content/fabc9042.png
            [type_id] => 18
            [sub_type_id] => 6
            [price_last_changed] => 2012-12-03 08:30:49 UTC
            [max_offer_unit_price] => 3180000
            [min_sale_unit_price] => 3890000
            [offer_availability] => 5574
            [sale_availability] => 6
            [gw2db_external_id] => 63505
            [sale_price_change_last_hour] => 0
            [offer_price_change_last_hour] => 0
        )

If anyone can point me in the right direction that would be awesome! I have searched for days and days but can't figure it out... I bet you it's really simple too :(

Answer 1

First of all, it would be good to know if there is a way to retrieve from the GW2 API server only the UPDATED items since your last "fetch all". This would mean that you would only need to retrieve this entire array once, and then update the few records that are updated each time.

1. You can split the array using something like array_chunk http://php.net/manual/en/function.array-chunk.php
2. Use these split arrays to generate batch insert statements for a database such as MySQL.
3. Use a simple SELECT query to retrieve the statements.

If you want to do it without DB you'll just have to search for the item using a for-loop...

for($i = 0; $i < $array['count']; $i++)
{
    if ($array['results'][$i]['name'] == 'Dusk')
     // you found the item
}

Answer 2

Try this.

Here you have used to create a new array and access it.

ini_set('max_execution_time', 0);
$data = json_decode(file_get_contents("http://www.gw2spidy.com/api/v0.9/json/all-items/all"), true);
$new_arrry = array();
$i = 0;
foreach($data as $item) {
    $new_arry[$i] = $item['name'];
    $new_arry[$i] = $item['data_id'];
    $i++;
}

Answer 3

Try a recursive array search. Here is an example using a recursive function posted on the php.net page for array_search. Note the function doesn't actually use the array_search php function, but provides a similar functionality which supports multi-dimensional arrays.

First the function:

function search($array, $key, $value)
{
    $results = array();

    if (is_array($array))
    {
        if (isset($array[$key]) && $array[$key] == $value)
            $results[] = $array;

        foreach ($array as $subarray)
            $results = array_merge($results, search($subarray, $key, $value));
        }

    return $results;
}

Now use the function:

$data = json_decode(file_get_contents("http://www.gw2spidy.com/api/v0.9/json/all-items/all"), true);
$results = $data['results'];
$foundItems = search($results, 'name', 'Dusk');
print_r($foundItems);

This should give you an array of all the items in the results that match the name. Similarly a search on id would be:

$foundItems = search($results, 'data_id', '12345');

Answer 4

Preferred Solution

Use a Database like MySQL to store all the request and only update the database if the JSON as been modified Why ?

$url = "http://www.gw2spidy.com/api/v0.9/json/all-items/all";
var_dump(get_headers($url));

Output

array (size=13)
  0 => string 'HTTP/1.1 200 OK' (length=15)
  1 => string 'Server: cloudflare-nginx' (length=24)
  2 => string 'Date: Mon, 03 Dec 2012 11:52:41 GMT' (length=35) <--- See Date
  3 => string 'Content-Type: application/json' (length=30)
  4 => string 'Content-Length: 8621869' (length=23)
  5 => string 'Connection: close' (length=17)
  6 => string 'cache-control: no-cache' (length=23)
  7 => string 'X-Varnish-TTL: 300.000' (length=22)
  8 => string 'Accept-Ranges: bytes' (length=20)
  9 => string 'Age: 135' (length=8)
  10 => string 'X-Varnish-Cache: HIT' (length=20)
  11 => string 'Set-Cookie: __cfduid=d249da7538d37614f7bc206fd407142eb1354535561; expires=Mon, 23-Dec-2019 23:50:00 GMT; path=/; domain=.gw2spidy.com' (length=133)
  12 => string 'CF-RAY: 286a7d677b004f85' (length=24)

As your can see it was last modified Date: Mon, 03 Dec 2012 11:52:41 GMT it does not make sense for you to pull all this information en every request

$url = "http://www.gw2spidy.com/api/v0.9/json/all-items/all";
$info = get_headers($url,true);

$lastDate = new DateTime();
$lastDate->setDate(2012, 12, 03); 

$dateModified = DateTime::createFromFormat("D, d M Y g:i:s \G\M\T", $info['Date']);
if($dateModified > $lastDate)
{

    //Parse Jason
    //Update Database etc 
}


// Select * from result where name = 'WHAT YOU WANT' 

*Educational Purpose *

$filter = new \ProductIterator("http://www.gw2spidy.com/api/v0.9/json/all-items/all");
$filter->find("name", "dusk"); //<---- What you want to find 

echo "<pre>";
foreach ( $filter as $value ) {
    var_dump($value);
}

Output (2 results Found)

Dusky Dye - 20428
Dusk Dye - 20564
Dusk - 29185

Total Time

1.4957299232483  <------------------ Total Time 

See Live Demo

Class Used

class ProductIterator extends FilterIterator {
    private $key;
    private $value;

    public function __construct($url) {
        $data = json_decode(file_get_contents($url), true);
        parent::__construct(new ArrayIterator($data['results']));
        unset($data);
    }

    public function find($key, $value) {
        $this->key = $key;
        $this->value = $value;
    }

    public function accept() {
        $user = $this->getInnerIterator()->current();
        if (isset($user[$this->key])) {
            return stripos($user[$this->key], $this->value) !== false;
        }
        return false;
    }
}