0

I need to export 20 million data in a pdf file. I have used the "WkHtmlToPdf" package (https://github.com/mikehaertl/phpwkhtmltopdf) in my yii2 application. I have used Mongo database, I have created a report module in which I need to export the pdf with large data. Code is working fine but it is taking too much time to write 1,00,000 it takes approx 10 mins so for 20 million approx it takes 20 hours so we are looking for a solution to write 20 million data.

Below is my query to fetch data from mongo:

$query = CannedReport::find()->where(['isParent'=>1]);
    $this->load($params);
    if (in_array(Yii::$app->user->identity->user_type, ['tenant', 'subtenant'])) {
        $query->andWhere(['tm_id' => (string)Yii::$app->user->identity->tm_id]);
        // $match['tm_id'] =(string)Yii::$app->user->identity->tm_id;
    }
    $query->andWhere(['!=', 'callstatus', 'completed']);
    $query->andWhere(['hangup' => 'ORIGINATOR_CANCEL']);

    $dataProvider = new ActiveDataProvider([
        'query' => $query,
        'pagination' => false,
        'sort' => [
            'defaultOrder' => ['start_epoch' => SORT_DESC],
        ],
    ]);

    if (isset($this->em_id) && !empty($this->em_id)) {
        $query->andFilterWhere(['in', 'em_id', $this->em_id]);
    }
    foreach (['start_epoch', 'end_epoch'] as $epoch) {
        if ($epoch == 'start_epoch') {
            if (isset($this->start_epoch) && !empty($this->start_epoch)) {
                $date = explode(' - ', $this->start_epoch);
            } else {
                $date[0] = date('Y-m-d H:i:s', strtotime('today - 31 days'));
                $date[1] = date('Y-m-d') . " 23:59:59";
                $this->start_epoch = implode(' - ', $date);
            }
        } else {
            $date = explode(' - ', $this->{$epoch});
        }
        if (isset($date[0]) && isset($date[1]) &&
            Yii::$app->helper->validateDate($date[0], 'Y-m-d H:i:s') && Yii::$app->helper->validateDate($date[1],
                'Y-m-d H:i:s')
        ) {
            $query->andFilterWhere([
                '>=',
                $epoch,
                (string)Yii::$app->helper->currentTimezone($date[0], 'Y-m-d H:i:s')
            ]);
            $query->andFilterWhere([
                '<=',
                $epoch,
                (string)Yii::$app->helper->currentTimezone($date[1], 'Y-m-d H:i:s')
            ]);
        }
        unset($date);
    }
    return $dataProvider;

Pdf generation code

$pdf = new Pdf(array(
        'binary' => '/usr/local/bin/wkhtmltopdf',
        'ignoreWarnings' => true,
        'commandOptions' => array(
            'useExec' => true,      // Can help on Windows systems
            'procEnv' => array(
                'LANG' => 'en_US.utf-8',
            ),
        ),
    ));

    $options = array(
        'dpi' => 96,
        'image-quality' => 100,
        'disable-smart-shrinking',
        'no-outline',            
    );
    $pdf->setOptions($options);
    $pdf->addPage($tableHtml);
    $detailHeadingHtml = '<div class="row" style="padding-top: 5px;"><h3 style="text-align: center; padding: 5px;">Detailed Records-Call Direction Report</h3><div>';
    $pdf->addPage($detailHeadingHtml . '' . $abandonedCallsHtml . '' . $completedCallsHtml . '' . $failedCallsHtml . '' . $transferredCallsHtml . '<table><tr><td align="center"><b>Total Count:</b></td><td align="center"><b>' . $dataProvider->query->count() . '</b></td></tr></table>');

   if (!$pdf->send($fileName)) {
        $error = $pdf->getError();
        echo $error;
    }
    exit();

Is there any optimized way to create a pdf with such large data using yii2 and mongo?

p1992
  • 31
  • 7
  • 2
    Is there any issue with this code? – Ravi Damasiya Nov 09 '21 at 11:01
  • are you getting out of memory? – Angel Deykov Nov 09 '21 at 11:05
  • @RaviDamasiya Code is working but it utilizes a high CPU, on exporting large data files so need to reduce that – p1992 Nov 09 '21 at 11:43
  • @AngelDeykov No, but it takes too long to export the pdf for large data. I also thought to break down file into chunks but still it is taking too much time – p1992 Nov 09 '21 at 11:49
  • Instead of one thing with 20 million records, try 20 things with 1 million records, and then combine those PDFs. Or break that down even further maybe, 2000 things with 10,000 records. Giant table rendering is an expensive operation even for modern browsers. – Chris Haas Nov 09 '21 at 12:53
  • @ChrisHaas Is there any way/tool/method to break down the pdf and then combine them into a zip file? – p1992 Nov 10 '21 at 05:20
  • You can either [zip](https://stackoverflow.com/a/4914894/231316) them or [combine them into a single PDF](https://stackoverflow.com/q/4794435/231316) – Chris Haas Nov 10 '21 at 13:02

0 Answers0