1

I have a foreach in cakephp that processes products from a distributor, but the thing is the lists have up to 200products each product can have 3 big pictures with 2 resizes. So i have in total 1200 big actions to much for one request. I breaked the foreach at each 10 products, removing them from the array and redirected to the same page. But after a while I get a redirect loop.

Any ideeas on how to avoid this? If I add another page in this redirect freenzy will it work? The redirect loop appears only when redirecting in the same page?

The thing is the loop will end, but the browser doesn't know that.

$this->data = $this->Session->read('Parser.data');
$limit = 0;
foreach ($this->data as $key => $data):
   $limit++;
   if ($limit == 4)
      $this->redirect($this->here);

   ...
   $this->Session->delete('Parser.data.' . $key);
endforeach;
$this->redirect(array('controller' => 'parser', 'action' => 'index')); //if $this->data is empty it redirects to upload page

The server work with any number of records from what I have tested, but I have this action along the lines:

$this->getImage(WWW_ROOT . $folder . DS, $new_path, $image['path']);

which looks like this:

protected function getImage($folder = null, $path = null, $from = null) {
    if (isset($from) && !empty($from))
        file_put_contents($folder . $path, file_get_contents($from));
}

this loads up the server's memory and crashes. This is why I have to break the foreach a couple of times. I also tried other functions to get the images as cUrl, but with same results!

amstegraf
  • 597
  • 1
  • 5
  • 11
  • can you show some code? – Nunser Dec 13 '13 at 17:33
  • Why are you redirecting so many times? You normally process data, and display it. May be your are calling many `redirects` many times. Show some code – Leonardo Dec 13 '13 at 17:34
  • Two thoughts: 1) depending on how many redirects you are able to trigger before the browser detects a loop (fewer redirects = longer process time per redirect), increase the # of products you process before redirecting 2) create a progress bar for the user and just process all the products that you have to in a single request while showing the user the the progress being made – iso27002 Dec 13 '13 at 18:45

1 Answers1

1

Let me copy my answer from another very similar question:

Never use URLs to do these kind of tasks, it is simply plain wrong, insecure and can cause your script to die or the server to become not responding any more.

Lets say you have 10000 users and a script runtime of 30 sec, it is very likely that the script times out before it finished and you end up with just a part of your users being processed at this time. The other scenario with a high or infinite amount of script runtime can lock your server. Depending on the script or DB actions it might cause the server to have a high load and users who use the site while the script is running will encounter a horrible slow to non responding site.

Also you can't really run a loop on a single URL, well you could redirect from one to another that does the limit and offset thing to simulate a loop over the 100000 users. If you don't loop over the records but fetch all 100000 at the same time it's likely your script dies because of running out of memory.

You should create a shell that processes the users in a loop and always just processes batches of for example 10, 50 or 100 users.

When executing your shell I recommend to use it with the "nice" command together to limit the amount of CPU time the shell is allowed to use to prevent the shell from taking 100% CPU usage to keep your site responding.

Look at creating a shell and setting up a cron in cake.

Community
  • 1
  • 1
floriank
  • 25,546
  • 9
  • 42
  • 66
  • I know I should create a shell and cron. I worked with this type of usage in other projects with more than 4000+ products per batch without problems. But in this project one administrator has to upload the file that is processed for import and change some configuration values. Thus it has to work in browser at request. And when doing so the image download action that works with php function file_get_contents and file_put_contents (I also tried with cUrl, but had the same result) freezes the server, because of memory usage! – amstegraf Dec 16 '13 at 14:11
  • And what exactly is the reason you can't use a cron and shell? – floriank Dec 16 '13 at 14:42
  • because the data is written on the fly, based on the file imported. After the import a user changes values of some fields and afterwards orders the process of the data. – amstegraf Dec 16 '13 at 14:49
  • i could use a shell for the process ... not a cron. Should I try this? – amstegraf Dec 16 '13 at 14:54
  • I'm doing something similar for video uploads. I have one shell running all time in a loop checking for new uploads. When it detects there is an upload it processes the file, flags the video as "in progress" and when it's done it flags it as "processed". You could do something similar. Have one shell detecting uploads that triggers another shell when an upload was detected to process the uploaded data. – floriank Dec 16 '13 at 15:24
  • this would be the same if I put the cron to run at 00:00 hours. Even if a process downloads the file just before, or a user uploads it during the day. I've done so in other projects, with no problems. the problem now is that the data is changed on the fly as previously stated, thus not allowing me to properly separate the data in to batches. In previous projects I had a process that reads the entire data and then sends each row individualy to be written to another action, waiting for return of status and then continuing. Avoiding any problems! But now I need something to work with one request! – amstegraf Dec 16 '13 at 15:30
  • 1
    This would not be the same as the cron is just triggered one time per day while in the other case the shell is checking for changes and processes them as they come in. Seriously, try it with a shell. I think my answer give enough reasons why a shell is superior in ever case. As a bonus you can get even a progress bar, estimate remaining time and send the shell output to a log. Also the footprint of the app is much smaller because you just load a shell and a few models instead of the whole MVC stack on each request. – floriank Dec 16 '13 at 15:35