Image migration script - Allowed memory size of 134217728 bytes exhausted

Question

I am trying to import around 130k images via url. I thought it would be convenient to write a cli artisan command.

I am using laravel and "Spatie/MediaLibrary".

This is my script:

public function handle()
    {
       Article::where('index_block_position',0)->each(function($article){
            if($article->featured_image != ''){
                try{
                    $featuredImage = $this->mediaLibrary->addMediaFromUrl($article, 'http://myoldwebsite.com' . $article->featured_image, 'featured_image');
                    $libraryImage = $this->mediaLibrary->addMediaFromUrl(MediaLibrary::first(), 'http://myoldwebsite.com' . $article->featured_image);
                    $article->index_block_position = 1;
                    $article->save();
                    $this->info("Article {$article->id} image: {$featuredImage->id} - Library image: {$libraryImage->id} - added");
                    Log::useDailyFiles(storage_path().'/logs/featured-image-migration.log');
                    Log::info("Article {$article->id} image: {$featuredImage->id} - Library image: {$libraryImage->id} - added");
                }catch (\Exception $e){
                    $this->info("image: {$article->featured_image} - exception: {$e->getMessage()}");
                    Log::useDailyFiles(storage_path().'/logs/featured-image-migration.log');
                    Log::info($e->getMessage());
                }
            }
        });
        $this->info('All done!');
    }

Unfortunately, after 10 or so images added I get the following error:

Symfony\Component\Debug\Exception\FatalErrorException: Allowed memory size of 134217728 bytes exhausted (tried to allocate 24576 bytes)

I am not sure how i can make this work because i have 130k images to import and the memory limit of 128M is already exhausted after 10 images...

I would appreciate suggestions.

score 0 · Answer 1 · answered Mar 09 '17 at 17:06

If you are using Laravel, consider breaking the upload down into jobs which can then be queued and processed over time.

https://laravel.com/docs/5.4/queues

You may also run into file upload limits set in your /etc/php/x.x/fpm/php.ini. More specifically these settings:

upload_max_filesize
post_max_size

Image migration script - Allowed memory size of 134217728 bytes exhausted

1 Answers1