0

I have a PHP application that returns forecasts values to a client in a json format. I hold all the data as i am collecting more in a variable.

If the client only requests values from a short period of time, life is good and he gets what he asked for. The problem is when he requests 1 or more years of forecasts. The exaust memory errors appears because the dataset stored in the variable is huge.

I don't want to extend memory, im thinking in alternative ways of doing it.

I came across with memory mapped file. Is that a good solution or you can think of others? Thank you!

Im using gearman and for each data retrieved by a worker a new array with values is added to my forecast variable.

Once all is finished i return it to client.

public function requestCompleteApi(\GearmanTask $task)
{                               
    $this->forecast[] = unserialize($task->data());
}
Andre Garcia
  • 894
  • 11
  • 30
  • 1
    won't help. if you run out of memory holding the raw data, you'll definitely run out of memory while the json string is being built. json isn't as "bloated" as xml and the like, but your average array WILL take up more space as a json string than it would going by the raw byte size of the data. – Marc B Jul 13 '16 at 16:39
  • Do you need the client to store that data or just display it? If it's just displaying you want I would try pagination, where every page shows about 50 results. – vlatkozelka Jul 13 '16 at 16:56
  • Yes, the client will need the data to be processed elsewhere. – Andre Garcia Jul 13 '16 at 16:58
  • @AndreGarcia Then you can use something like http://stackoverflow.com/questions/6914912/streaming-a-large-file-using-php. You could collect the data chunk by chunk in that variable , and flush that variable so the client receives what is getting buffered. And maybe change the content type in the header to plain text. Also don't forget to set time limit to 0 so it wouldn't time out – vlatkozelka Jul 13 '16 at 17:02

1 Answers1

0

So, it very vague question, but I try to help.

I have some sugastes:

  1. Try use PHP 7 - it use less memory to store variables.
  2. If you use big arrays try reformat you application. Maybe you can use generators (in PHP 5.5+) PHP Generators
  3. Maybe you can use less memory if try not JSON but more compact formats like ProtoBuf or Apache Thrift
Lakremon
  • 787
  • 1
  • 8
  • 26
  • Im using Gearman, not supported in php 7. For now php 7 is not an option. As PHP Generators and Apache Thrift both look promising! I will look into it! Thanks! – Andre Garcia Jul 13 '16 at 17:06
  • About Gearman - look also on Net Gearman [link]https://github.com/Publero/net_gearman, it don't need PHP Extansion. Maybe it help you use PHP 7. – Lakremon Jul 13 '16 at 17:57