Undefined index error when $em->clear() in Symfony3 / Doctrine2

Question

I am getting this error:

[Symfony\Component\Debug\Exception\ContextErrorException] Notice: Undefined index: 000000000fa82729000000006e17190b

I have tried all the tips from this post.

My code is much simplier than the codes of other posters (asking about this problem).

    $em = $this->getContainer()->get('doctrine')->getManager();
    $pictureUrlRepository = new ProductItemPictureUrlRepository($em);
    $pictures = $pictureUrlRepository->findAll();

    for ($i = 0; $i < 4; ++$i) {
        $productItem = $pictures[$i]->getProductItem();
        dump($productItem->getId());
        $fileName = 'file_' . $i;

        $productItemPicture = new ProductItemPicture();
        $productItemPicture->setProductItem($productItem);
        $productItemPicture->setFile($fileName);
        $productItemPicture->setPosition(1);
        $productItemPicture->setSize(1000);
        $productItemPicture->setWidth(1200);
        $productItemPicture->setHeight(1000);

        $em->persist($productItemPicture);
        $em->flush();
        $em->clear();
        unset($productItemPicture);
    }

To show you, that I have tried the advices from the link above, here's the more complex code with implemented tips (but the result is still the same):

      $em = $this->getContainer()->get('doctrine')->getManager();
    $pictureUrlRepository = new ProductItemPictureUrlRepository($em);
    $pictures = $pictureUrlRepository->findAll();

    foreach ($em->getEventManager()->getListeners() as $event => $listeners) {
        foreach ($listeners as $listener) {
            $em->getEventManager()->removeEventListener($event, $listener);
        }
    }

    $batchSize = 2;
    for ($i = 0; $i < 4; ++$i) {
        $productItem = $pictures[$i]->getProductItem();
        dump($productItem->getId());
        $fileName = 'file_' . $i;

        $productItemPicture = new ProductItemPicture();
        $productItemPicture->setProductItem($productItem);
        $productItemPicture->setFile($fileName);
        $productItemPicture->setPosition(1);
        $productItemPicture->setSize(1000);
        $productItemPicture->setWidth(1200);
        $productItemPicture->setHeight(1000);
        $productItemPicture->setCreated(new \DateTime('20.1.2017'));
        $productItemPicture->setUpdated(new \DateTime('20.1.2017'));

        $em->persist($productItemPicture);
        if (($i % $batchSize) === 0) {
            $em->flush();
            $em->clear();
        }
    }
    $em->flush();
    $em->clear();

Only when I comment (disable) $em->clear(); line, it works. What's the deal with not running the clear()? What can I expect when doing thousands of DB entries and not running clear()?

Could you please mark "the line" from which this error comes from? Maybe from code you are executing just after the loop? — Kamil Adryjanek, Jan 20 '17 at 09:14
Maybe but you are clearing `EntityManager`and if you are trying to update other entities later after loop, this will rise exception you provided. — Kamil Adryjanek, Jan 20 '17 at 09:25
@Veve I have thousands of pictures, but for the sake of testing, I reduced to 4, to show, that it's still causing this problem. Other threads involving this problem were about memory leaks, so I downsized this test to 4, to trigger this error. — František Heča, Jan 20 '17 at 09:38
@KamilAdryjanek I don't think I am updating other entities. From the code is visible I loop and update one entity, at least I hope so. — František Heča, Jan 20 '17 at 09:39
I see that but what I'm asking is are you using `EntityManager` later after the loop? — Kamil Adryjanek, Jan 20 '17 at 11:12
The error is being triggered already inside of this loop. It seems like in the second cycle. — František Heča, Jan 20 '17 at 12:33

score 1 · Answer 1 · answered Jan 23 '17 at 07:56

I have found what was the problem in my case.

Before I begin the itearation cycle, I get all results with FindAll() method. Then I set $productItem by referencing array results from that method. But the clear() in the iteration cycle destroys Entity IDs in that array.

When I was checking Doctrine website examples about batch processing, I was more concerned about the clear() part. But the real problem is how they get the results they iterate on. They use iterate() methods on createQuery() result.

So, the updated code, working result is this:

    $em = $this->getContainer()->get('doctrine')->getManager();
    $q = $em->createQuery('select p from AppBundle\Model\Product\Item\ProductItemPictureUrl p');
    $iterableResult = $q->iterate();

    $i = 0;
    foreach ($iterableResult as $row) {
        $productItem = $row[0]->getProductItem();
        dump($productItem->getId());
        $fileName = 'file_' . $i;

        $productItemPicture = new ProductItemPicture();
        $productItemPicture->setProductItem($productItem);
        $productItemPicture->setFile($fileName);
        $productItemPicture->setPosition(1);
        $productItemPicture->setSize(1000);
        $productItemPicture->setWidth(1200);
        $productItemPicture->setHeight(1000);

        $em->persist($productItemPicture);
        $em->flush();
        $em->clear();
        unset($productItemPicture);
        $i++;
    }

Thanks to @Cerad for pointing me into this direction.

For the reference, I have found this topic, which is not exactly the same issue, but brings some light to this problem: http://stackoverflow.com/questions/7431794/doctrine-2-weird-behavior-while-batch-processing-inserts-of-entities-that-refer — František Heča, Jan 31 '17 at 10:34

Cerad · Answer 2 · 2017-01-20T14:26:53.837

0

One basic problem is:

// Replace
$pictureUrlRepository = new ProductItemPictureUrlRepository($em);

// With
$pictureUrlRepository = $em->getRepository('AppBundle:ProductItemPictureUrl');

There is a complex relation between entity manager and it's repositories. Never try to instantiate a repository directly. Lots of stuff going on behind the scenes. Surprised that your findAll even worked.

Of course it also seems like you added doctrine event listeners? So it is hard to say exactly where the problem might be. If the repository fix does not work then post the complete stack trace.

You also seem to be relying on having at least four pictures per product which is a bit dangerous.

============================

Not really clear from your comments if the problem was solved with the repository suggestion. But using $em->clear at the same time you are relying on $pictures is a bit dangerous as well. No telling what is happening to the entities when you clear the manager. So move them outside the loop:

for ($i = 0; $i < 4; ++$i) {
    $em->persist($productItemPicture);
}
$em->flush();
$em->clear();

And yes I understand that you are probably reaching memory limits when your process large number of items. The ORM is not very good at large batch jobs. You might need to drop down to using the DBAL connection object.

edited Jan 20 '17 at 14:26

answered Jan 20 '17 at 13:31

Cerad

48,157
8
90
92

Thank you for interest. I understand your concern about the "new" part with repositories. I am new with Symfony and I added this "new" part only because of posting code here to Stackoverflow. And the reason was, that I found similar code in other part of the project, coded by fellow programmer and that code works. So, I was myself using this code $doctrine->getRepository(ProductItemPictureUrl::class) which I hope is the same as you posted. – František Heča Jan 20 '17 at 14:16
I am not using event listeners. In my second code example, I unregister them only because previous answers in other thread suggested this, in case of some internal listener is present. – František Heča Jan 20 '17 at 14:17
The relying on count of pictures is also not true in my production code. This actual function is only for testing purposes, to track this bug. – František Heča Jan 20 '17 at 14:18
I am not sure how to get stack trace for this error, how to get more informations than the simple "Notice: Undefined index" message. – František Heča Jan 20 '17 at 14:22
Thanks for answers. The repository suggestion didn't solve the problem, it works the same. When I put the flush() and clear() outside of the loop, it's working ok. The same with not using clear() at all. Interesting is, that when I run my original code, with clear() inside of the loop, there's always 1 item inserted into DB table. The clear() is not causing the error directly. I suppose that only after the $em is used again (after that first clear()), it causes the error. So, when I move the clear(), outside of the loop, it's not triggering any error, because my script ends there. – František Heča Jan 20 '17 at 18:10
Actually, the problem is that setProductItem($productItem); requires $productItem to be in the unit of work object inside of the entity manager. Clearing the manager clears the product items. So it works for the first one because the clear has not been done and fails on the second one. – Cerad Jan 20 '17 at 18:14
thanks for comment, but where it is not set? You can see from the code, that I set $productItem properly at the beginning of each loop. – František Heča Jan 23 '17 at 06:09
It seems like you helped me to narrow possible problems Cerad. It seems like you can't iterate on results get by FindAll() and use clear() in the iteration cycles. The examples at Doctrine website for batch processing do use some iterate() functions instead. So I am now trying to find out, if it's still possible to somehow iterate on Find() function results, with flush(), clear() in the loops. – František Heča Jan 23 '17 at 07:22

Undefined index error when $em->clear() in Symfony3 / Doctrine2

2 Answers2