1

I'm copying contacts from propertyware to Podio using a cron job, and I've already copied contacts into the contacts.json file to reduce Podio API calls. However, after copying 2000+ contacts my JSON file empties and API starts coping from the start or first contact. And contacts are being copied in chunks of around 200 contacts each.

I'm unable to find the reason, why does my file get empty after some time?

Here is the code

if (!file_exists(PROPERTYWARE_CONTACTS_FILE_NAME)) {
    file_put_contents(PROPERTYWARE_CONTACTS_FILE_NAME, "[]");
}
$response = HttpClient(HTTP_GET, PROPERTYWARE_CONTACTS_URL);

$response = json_decode( preg_replace('/[\x00-\x1F\x80-\xFF]/', '', $response), true );
$saved_records = json_decode( preg_replace('/[\x00-\x1F\x80-\xFF]/', '', file_get_contents(PROPERTYWARE_CONTACTS_FILE_NAME)), true );

$incoming_records = array();
$different_from_saved = array();
$old_items = array();

foreach ($response as $key => $val) {
    if ($key == "records") {
        $incoming_records = $val;
    }
}

foreach ($incoming_records as $incoming_record) {
    $add_it = true;
    foreach ($saved_records as $saved_record) {
        if ($saved_record == $incoming_record) {
            $add_it = false;
            array_push($old_items, $incoming_record);
        }
    }
    if ($add_it) {
        array_push($different_from_saved, $incoming_record);

    }
}

if (count($different_from_saved) > 0) {
    print "\nDifferent Data";
}
updateInPodio($different_from_saved, CONTACTS_APP_ID, $old_items);



function updateInPodio($items, $app_id, $old_items)
{
    $old_items_count = count($old_items);
    $new_items_count = count($items);
    $pushed_into_podio = 0;
    $saveToFile = [];
    foreach ($old_items as $old_item) {
        array_push($saveToFile, (object)$old_item);
    }
    try {
        foreach ($items as $item) {
            $phones = [];
            $emails = [];
            $fields = [];
            $fields['propertyware'] = "YES";

            if(!empty($item[0]))
                $fields['entitiy-id'] = $item[0];
            if(!empty($item[1]))
                $fields['first-name'] = $item[1];
            if(!empty($item[2]))
                $fields['last-name'] = $item[2];
            if(!empty($item[3]))
                $fields['company'] = $item[3];

            $response = PodioItem::create($app_id, ["fields" => $fields]);
            $pushed_into_podio++;
            array_push($saveToFile, (object)$item);
        }
        file_put_contents(PROPERTYWARE_CONTACTS_FILE_NAME, json_encode($saveToFile));
    } catch (Exception $exception) {
        file_put_contents(PROPERTYWARE_CONTACTS_FILE_NAME, json_encode($saveToFile));
        print "<pre>";
        print_r($exception->getMessage());
        print "</pre>";

        $to = "abdulhaye111@gmail.com";
         $subject = "Exception While Pushing Contacts to Podio";

         $message = "<br><strong>OLD_CONTACTS:</strong> $old_items_count<br><strong>NEW_CONTACTS:</strong> $new_items_count<br><strong>PUSHED_CONTACTS:</strong> $pushed_into_podio<br><br><br>".$exception->getMessage();

         $header = "From:noreply@propertymanagementoh.com \r\n";
         $header .= "MIME-Version: 1.0\r\n";
        $header .= "Content-type: text/html\r\n";

         $retval = mail ($to,$subject,$message,$header);

         if( $retval == true ) {
            echo "Message sent successfully...";
         }else {
            echo "Message could not be sent...";
         }
    }
    file_put_contents(PROPERTYWARE_CONTACTS_FILE_NAME, json_encode($saveToFile));
    print"<br>OLD_ITEMS: $old_items_count<br>NEW_ITEMS: $new_items_count<br>PUSHED_ITEMS: $pushed_into_podio<br>";

    header("Location: http://propertymanagementoh.com/dev/building.php");
    header("Location: http://propertymanagementoh.com/dev/lease.php");
    header("Location: http://propertymanagementoh.com/dev/leasePayments.php");
    header("Location: http://propertymanagementoh.com/dev/protfolioBills.php");
    header("Location: http://propertymanagementoh.com/dev/units.php");

}
Abdul Haye
  • 47
  • 1
  • 9
  • 1
    Whats all this `header("Location: ` stuff. As far as your JSON goes, you could be running out of memory trying to open it, or there could be some other error that makes the existing json return false, I would add some proper error checking when decoding it, `json_last_error` etc. – ArtisticPhoenix Apr 19 '19 at 16:27
  • @ArtisticPhoenix these are the same scripts as contacts to copy buildings, leases, lease payments, portfolio bills, units, etc.... into Podio – Abdul Haye Apr 19 '19 at 16:29
  • `header("Location:` - is a redirect in PHP, and not using exit after each location call can cause undesirable behavior, it's just a bad Idea to redirect to 5 or 6 things in a row. – ArtisticPhoenix Apr 19 '19 at 16:30
  • @ArtisticPhoenix can you suggest me best way for redirection. Because client don't want to add each file separately in cron. He said call other scripts from your own code. – Abdul Haye Apr 19 '19 at 16:33
  • @ArtisticPhoenix And emptying JSON is the reason of these redirections? – Abdul Haye Apr 19 '19 at 16:34
  • `And emptying JSON is the reason of these redirections` - no these are 2 separate issues. `can you suggest me best way for redirection` - depends what they do, you probably don't want to redirect to them. You either want to call them in a new thread with like `shell_exec` etc. or include them in the current one. Probably what I would do is something like `exec('usr/bin/curl -s {url} &')` The `&` at the end creates a new background process depending on your hardware that may be too much at one time. If you leave that off they run sequentially. – ArtisticPhoenix Apr 19 '19 at 16:40
  • Not to mention this (or any) output will kill those header calls `print"
    OLD_ITEMS: $old_items_count
    NEW_ITEMS: $new_items_count
    PUSHED_ITEMS: $pushed_into_podio
    ";` so right now they do nothing but generate warning messages. Please see https://stackoverflow.com/questions/8028957/how-to-fix-headers-already-sent-error-in-php     – ArtisticPhoenix Apr 19 '19 at 16:45
  • @ArtisticPhoenix You're right sir. Thanks for this information, I'll be careful next time. Did you get the reason for emptying JSON file after a certain time? – Abdul Haye Apr 19 '19 at 16:56
  • Sure, to figure out the JSON, you'll want to trap errors after this line `$saved_records = json_decode(` so add something like `if(json_last_error()) file_put_contents('errors.txt', json_last_error_msg()."\n", FILE_APPEND)` Then when your file becomes empty, check to see if you have some kind of decoding error. A decoding error will make that `FALSE` Then a large part of your script will blowup with not checking on that variable (or other json encode/decode). That's just a guess, or you could be running into memory limits if you have large JSON files too. – ArtisticPhoenix Apr 19 '19 at 17:01
  • @ArtisticPhoenix I've added this if clause, it never true even once and JSON file gets empty again. I'm not able to find the root cause of this problem. here what I've done `if(json_last_error())` `file_put_contents('json_error.txt', json_last_error_msg()." \r\n", FILE_APPEND);` – Abdul Haye Apr 23 '19 at 20:27
  • Yes but you should do that after any JSON decode or encode call. More likely it will be decoding incoming JSON. Reading JSON you save is less likely to be an issue. Other then that I am not sure why it would be empty – ArtisticPhoenix Apr 24 '19 at 18:18
  • Let me explain, lets look at if `$response = json_decode(..)` returns false, such as a failure to decode. The `foreach ($response ` will issue a warning because it's boolean not an array, and then these 3 `$incoming_records,$different_from_saved,$old_items;` will be left in their initial state (an empty array). then you call `updateInPodio([],[],[])` which further results in `$saveToFile` being left as an empty array, which in the end results in the file being written with empty JSON. - this is why I suspect a JSON read error. – ArtisticPhoenix Apr 24 '19 at 18:24
  • In the above case you should also see `Warning: Invalid argument supplied for foreach` or similar if you have error reporting at the right level. One other possibility is if `$response = json_decode(..)` returns an empty array. Because the way the code is written it's required that the first foreach runs on some data. Or those again are all empty - resulting in an empty write to the file as there are no sanity checks for empty data. – ArtisticPhoenix Apr 24 '19 at 18:26
  • One simple thing to test / avoid is after `$response = json_decode(..)` add `if(!$response) exit()` or similar (log to file etc.) to avoid this case. Ideally the code should be written in a way that does not depend on new data. For example instead of adding to `$old_items` you could start with the saved items, then remove from that. (Subtraction) this way if it's empty no subtraction happens and no data changes. You also call this 3x `file_put_contents(PROPERTYWARE_CONTACTS_FILE_NAME, json_encode($saveToFile));` when one would do. 1 in the Try, 1 in the Catch and one after. – ArtisticPhoenix Apr 24 '19 at 18:33

0 Answers0