PHP Phar creation slow on a powerful PC, how to speed up (loading/reading ~3000 files)?

Question

I'm trying to pack my web application (Symfony 2 project) with Phar. I've successfully packed Silex, a micro framework with hundred of files in a reasonable time (1-2 minutes).

The problem is on my development machine (i7 4770k, 16GB, SSD Raid 0, project on a RAM disk) creating the archive is really slow, it takes ~1 sec for each file. I really need to find out a way to speed up things.

Single iteration of reading/loading the file is slow. I'm adding files using:

function addFile(Phar $phar, SplFileInfo $file)
{
    $root = realpath(__DIR__.DIRECTORY_SEPARATOR.'..'.DIRECTORY_SEPARATOR);
    $path = strtr(str_replace($root, '', $file->getRealPath()), '\\', '/');
    $phar->addFromString($path, file_get_contents($file));
}

$phar = new Phar(/* ... */);
$phar->startBuffering();

// ...    
foreach ($files as $file) {
    addFile($phar, $file);
}

// ...
$phar->setStub(/* ... */);
$phar->stopBuffering();

How can I speed up reading/adding files? Could be my OS (Windows) the problem?

EDIT: disabling buffering didn't solve the problem. Same speed adding from strings:

// This is VERY fast (~ 1 sec to read all 3000+ files)
$strings = array();
foreach ($files as $file) {
    $root = realpath(__DIR__.DIRECTORY_SEPARATOR.'..'.DIRECTORY_SEPARATOR);
    $path = strtr(str_replace($root, '', $file->getRealPath()), '\\', '/');

    $strings[$path] = file_get_contents($file->getRealPath());
}

// This is SLOW
foreach ($strings as $local => $content) {
    $phar->addFromString($local, $content);
}

EDIT: full quick&dirty script (may help) app/build:

#!/usr/bin/env php
<?php

set_time_limit(0);

require __DIR__.'/../vendor/autoload.php';

use Symfony\Component\Finder\Finder;
use Symfony\Component\Console\Input\ArgvInput;

$input = new ArgvInput();
$env = $input->getParameterOption(array('--env', '-e'), 'dev');

function addFile(Phar $phar, SplFileInfo $file)
{
    $root = realpath(__DIR__.DIRECTORY_SEPARATOR.'..'.DIRECTORY_SEPARATOR);
    $path = strtr(str_replace($root, '', $file->getRealPath()), '\\', '/');
    $phar->addFromString($path, file_get_contents($file));
}

$phar = new Phar(__DIR__ . "/../symfony.phar", 0, "symfony.phar");
$phar->startBuffering();

$envexclude = array_diff(array('dev', 'prod', 'test'), array($env));

// App
$app = (new Finder())
    ->files()
    ->notPath('/cache/')
    ->notPath('/logs/')
    ->notName('build')
    ->notname('/._('.implode('|', $envexclude).')\.yml$/')
    ->in(__DIR__);

// Vendor
$vendor = (new Finder())
    ->files()
    ->ignoreVCS(true)
    ->name('*.{php,twig,xlf,xsd,xml}')
    ->notPath('/tests/i')
    ->notPath('/docs/i')
    ->in(__DIR__.'/../vendor');

// Src
$src = (new Finder())
    ->files()
    ->notPath('/tests/i')
    ->in(__DIR__.'/../src');

// Web
$web = (new Finder())
    ->files()
    ->in(__DIR__.'/../web')
    ->notname('/._('.implode('|', $envexclude).')\.php$/');

$all = array_merge(
    iterator_to_array($app),
    iterator_to_array($src),
    iterator_to_array($vendor),
    iterator_to_array($web)
);

$c = count($all);
$i = 1;
$strings = array();
foreach ($all as $file) {
    addFile($phar, $file);
    echo "Done $i/$c\r\n";
    $i++;
}

$stub = <<<'STUB'
Phar::webPhar(null, "/web/app_phar.php", null, array(), function ($path) {
    return '/web/app_phar.php'.$path;
});

__HALT_COMPILER();
STUB;

$phar->setStub($stub);
$phar->stopBuffering();

Answer 1

Try using Phar::addFile($file) instead of Phar::addFromString(file_get_contents($file))

i.e.

function addFile(Phar $phar, SplFileInfo $file)
{
    $root = realpath(__DIR__.DIRECTORY_SEPARATOR.'..'.DIRECTORY_SEPARATOR);
    $path = strtr(str_replace($root, '', $file->getRealPath()), '\\', '/');
    //$phar->addFromString($path, file_get_contents($file));
    $phar->addFile($file,$path);
}

Answer 2

Phar::addFromString() or Phar:addFromFile() is incredibly slow. Like @sectus said Phar::buildFromDirectory() is a lot faster. But as an easy alternative with little changes to your code you could use Phar::buildFromIterator().

Example:

$all = $app->append($vendor)->append($src)->append($web);
$phar->buildFromIterator($all, dirname(__DIR__));

instead of:

$all = array_merge(
    iterator_to_array($app),
    iterator_to_array($src),
    iterator_to_array($vendor),
    iterator_to_array($web)
);

$c = count($all);
$i = 1;
$strings = array();
foreach ($all as $file) {
    addFile($phar, $file);
    echo "Done $i/$c\r\n";
    $i++;
}

$ time app/build

real    0m4.459s
user    0m2.895s
sys     0m1.108s

Takes < 5 seconds on my quite slow ubuntu machine.

Answer 3

I'd suggest to dig in php config.

First recomendation - is to disable open_basedir if it is enabled. As i understand php internals, when u try to access any file location with php, it is a must for php to check if file location matches allowed directory-tree. So if there are many files this operation will be preformed for each file, and it can slow the proccess down significantly. On the other hand if open_basedir is disabled, check is never done.

http://www.php.net/manual/en/ini.core.php#ini.open-basedir

Second - is to check realpath_cache_size and realpath_cache_ttl.

As written in php description

Determines the size of the realpath cache to be used by PHP. This value should be increased on systems where PHP opens many files, to reflect the quantity of the file operations performed.

http://www.php.net/manual/en/ini.core.php#ini.realpath-cache-size

I hope this will help u to speed up ur operations.

Answer 4

You know, I have realized that Phar::buildFromDirectory is pretty fast.

$phar->buildFromDirectory('./src/', '/\.php$/');

But you need write more complicated regex. But you could call buildFromDirectory several times with different arguments.

Or create temporary folder and copy all files into from $all. Something like this

function myCopy($src, $dest)
{
    @mkdir(dirname($dest), 0755, true);
    copy($src, $dest);
}

foreach ($all as $file)
{
    //$phar->addFile($file);
    myCopy($file, './tmp/' . $file);
}

$phar->buildFromDirectory('./tmp/');

Answer 5

I would suggest using Preloader to concatenate all your files into a single file and then simply add that single file to the phar.

Answer 6

I know you said that adding the filename from string did not improve performance, but perhaps a different way of loading the file names can improve performance along with using the filename from string. Composer is pretty fast, but I never timed it. Try loading the files by group of filetypes and adding them as groups separately.

It uses a class from Symfony which you may not want or would change.

use Symfony\Component\Finder\Finder;

$phar = new \Phar(/* ... */);
$phar->setSignatureAlgorithm(\Phar::SHA1);
$phar->startBuffering();
$finder = new Finder();

//add php files with filter
$finder->files()
        ->ignoreVCS(true)
        ->name('*.php')
        ->notName('Compiler.php')
        ->notName('ClassLoader.php')
        ->in(__DIR__.'/..')
    ;

foreach ($finder as $file) {
    $this->addFile($phar, $file);
    }

$this->addFile($phar, new \SplFileInfo(/* ... */), false);

$finder = new Finder();
$finder->files()
     ->name('*.json')
    ->in(__DIR__ . '/../../res')
    ;

foreach ($finder as $file) {
     $this->addFile($phar, $file, false);
    }

    $this->addFile($phar, new \SplFileInfo(/* ... */), false);

$phar->setStub($this->getStub());
$phar->stopBuffering();

Perhaps you can exclude a cache or log file using the Finer's filters if somehow its one large file that causes the long lag. Look at the composer link for full details on how its implemented.

Answer 7

what about using class instead of passing phar into func? just a piece of code to understand.. or ever heard about memory limit of php.ini or other setting that can slow down things.

class XY {

private $phar;

function addFile(SplFileInfo $file)
    $root = realpath(__DIR__.DIRECTORY_SEPARATOR.'..'.DIRECTORY_SEPARATOR);
    $path = strtr(str_replace($root, '', $file->getRealPath()), '\\', '/');
    $this->phar->addFromString($path, file_get_contents($file));
}
// other code here
}

Correct me if i am wrong but in this way instead of passing the phar into function, you will avoid "copying" of object. this way is like pointer.

Answer 8

you can create threads and reduce total time, of course Symfony has to support concurrent loading. it is not actually best answer for your question, but it can significantly decrease total load time.

class Loader extends Thread {
    public $phar;
    public $file;
    public $done;
    public function run() {
        $self->done = false;
        addFile($self->phar, $self->file);
        $self->done = true;
    }
}
$threads = array();
foreach ($files as $file) {
    $t = new Loader();
    $t->phar = $phar;
    $t->file = $file;
    addFile($phar, $file);
    $t->start();
    $threads[] = $t;
}
while(true){
  $finished = true;
  foreach ($t as $threads) {
     if ($t->done == false){
        $finished = false;
        sleep(1);
        break;
     }
  }
  if ($finished)
    break;
}

and, creating 3000 threads is not good idea. you might need to create well thread-worker logic.

Answer 9

Try to disable GC like composer did

gc_disable();