0

I am parsing 2 JSON files and saving only some necessary data into MySQL database right away. Problem is in execution time of this script, since there is pretty much over 1000 games in the json file.

What the script does is that it parses first json file which is a list of games, where only some of those games are possible to get embed code from.

Steps to get what I want:

  1. Parse $game_list_url
  2. Decode and save game_id into $game_id

Now, since I got $game_id from $game_list_url file, I can parse another file, which is a specific json file of the game I want to get info about - info whether it has embed code which is final target of this entire script. A way of finding out if the game's json file has embed v variable is to check if the $webmaster_game == 1. If it is, thats the game I want!

So moving on from step 2, the next steps would be:

  1. Parse JSON file of a specific game --> $get_game_by_url
  2. Decode and save webmaster_game into $webmaster_game

  3. Check if $webmaster_game == 1

    a) if not, break out of inner cycle and go for next game_id

    b) if yes, decode and store other info into variables and execute() query

This script takes too long to execute - it runs over max_execution_time. What could dramatically simplify the complexity of this or how could I change it so it doesn't take so much time?

EDIT

Regarding to suggested answer as duplicate, extending php_execution_time does not solve my problem, since I have already done it and it does NOT simplify complexity of my script.

I am looking for a way to reduce the execution time to shortest possible. Can saving the json files save some time, are there any other options?

See my code below

<?php

include './db/connect.php';

$game_list_url = 'https://webmasters.miniclip.com/api/games/en.json';
$data = file_get_contents($game_list_url);
$json = json_decode($data,true);

$stmt = $conn->prepare("INSERT INTO game     (game_id, name, main_category_id, 
                                                    width, height, embed,
                                                    big_icon_path, small_icon_path
                                                    ) 
                                                    VALUES (?, ?, ?, ?, ?, ?, ?, ?)");

foreach ($json as $row){
    //decode json vars to php vars
    $game_id            = $row['game_id'];

    //parse game by game_id to check if it is possible to get embed code of the game
    $get_game_by_url = 'https://webmasters.miniclip.com/api/games/'.$game_id.'/en.json';
    echo $get_game_by_url;
    echo "\n";
    $data2 = file_get_contents($get_game_by_url);
    $json2 = json_decode($data2,true);

    foreach ($json2 as $row2){
        $webmaster_game     = $row2['webmaster_game'];
        //if webmaster_game != 1 then skip this game, because we cannot get embed code from miniclip
        if($webmaster_game != "1")
            break;
        $small_icon_path    = $row2['small_icon'];
        $big_icon_path      = $row2['big_icon'];
        $name               = $row2['name'];
        $main_category_id   = $row2['main_category_id'];
        $width              = $row2['width'];
        $height             = $row2['height'];
        $embed              = $row2['embed'];

        $stmt->bind_param("isiiisss",     $game_id, $name, $main_category_id,
            $width, $height, $embed,
            $big_icon_path, $small_icon_path
        );
        $stmt->execute();
    }
}

//script done, close all connections
$stmt->close();
$conn->close();
?>
Ady96
  • 686
  • 4
  • 12
  • 35
  • why not use cache？ – winlans Mar 05 '18 at 23:45
  • I have never used cache, this is also my first time with parsing `json` files..any "simple" solution which will do what I want will be okay with me – Ady96 Mar 05 '18 at 23:48
  • I would bet that most of the time in this script is taken up making the call to `webmasters.miniclip.com` in the loop – RiggsFolly Mar 05 '18 at 23:49
  • @winlans I am interested, use what cache? – RiggsFolly Mar 05 '18 at 23:49
  • @RiggsFolly so what you want to suggest is download the file first and then parse it? Or as winlans mentioned cache, maybe caching the file would help with faster file reading? – Ady96 Mar 05 '18 at 23:50
  • Not quite sure what you mean by that – RiggsFolly Mar 05 '18 at 23:52
  • According to their API docs, `https://webmasters.miniclip.com/api/genre/432/en.json` will get you only the webmasters games. But if you want to get the actual embed code for each one, you'll still have to make a separate API request for each game, which will still be slow. But not AS slow. – Don't Panic Mar 05 '18 at 23:53
  • @Don'tPanic well that's what I am doing right now, isn't it? I am parsing list of all games, and then I parse every game's file separately. – Ady96 Mar 05 '18 at 23:55
  • Do you run this code very often? – RiggsFolly Mar 05 '18 at 23:56
  • Yes, it will just let you eliminate the `if($webmaster_game != "1")` bit, because that will return only the webmaster games rather than all the games. – Don't Panic Mar 05 '18 at 23:56
  • btw, with your current code, `if($webmaster_game != "1") break;`, you probably want `continue` rather than `break`, otherwise it will stop the loop the first time it encounters a non-webmaster game. – Don't Panic Mar 05 '18 at 23:57
  • @RiggsFolly well, I will run this code only when I will want to update my database for new games, pretty much. When I get all data I want into DB, no need to parse over and over. – Ady96 Mar 05 '18 at 23:57
  • @Don'tPanic it runs fine now, it did not stop on first non-webmaster game. – Ady96 Mar 05 '18 at 23:59
  • @RiggsFolly maybe file cache is fit for his. – winlans Mar 06 '18 at 00:01
  • @winlans So where are we going to find the mythical cache full of the info we are interested in? – RiggsFolly Mar 06 '18 at 00:02
  • Define _Script takes to long to run_ Do you mean it blows the `max_execution_time` – RiggsFolly Mar 06 '18 at 00:06
  • @RiggsFolly yes, it does. I have added that into my question just now. – Ady96 Mar 06 '18 at 00:08
  • @RiggsFolly http://symfony.com/doc/current/components/cache.html – winlans Mar 06 '18 at 00:09
  • 1
    @winlans 1) Does this look like Symfony code to you? 2) You have to load a cache, they are not magic! So OP would have to make all the calls to the remote server anyway to load the cache with anything useful. – RiggsFolly Mar 06 '18 at 00:12
  • I have tried to edit the first file as @Don'tPanic mentioned, it did go much faster than before, but it still takes about 40 seconds to save 270 games. – Ady96 Mar 06 '18 at 00:17
  • 1
    **Are we worrying about a script that you run once in a while that takes 40 second to run????** When I say we, I mean everyone else, I no longer care – RiggsFolly Mar 06 '18 at 00:21
  • 1
    _but it still takes about 40 seconds to save 270 games_ **No!** It probably takes 38 seconds to ask for and be given the data by the remote website – RiggsFolly Mar 06 '18 at 00:24
  • @RiggsFolly I meant it takes 40 seconds to finish the process. And this is not a duplicate, since I am asking about simplifiying complexity, not how to extend `php_execution_time`, since I have already done it. – Ady96 Mar 06 '18 at 07:58
  • It seems that your code currently works, and you are looking to improve it. Generally these questions are too opinionated for this site, but you might find better luck at [CodeReview.SE](//codereview.stackexchange.com/tour). Remember to read [their requirements](//codereview.stackexchange.com/help/on-topic) as they are a bit more strict than this site. – Blue Mar 09 '18 at 17:42

0 Answers0