0

I have created an application that uses the Archon library for creating and manipulating dataframes, here's the link(Great Library btw!).

The application will create a large array that contains duplicates. I am using the dataframe functionality of Archon to perform a groupBy operation, so that I get the count of each unique entry to the array.

This array varies from being a couple of hundred entries, to many thousand.

Locally, this works fine. I'm using Vagrant, with VirtualBox - and I have a Homestead Box running there. It is running on an Ubuntu 64 bit system. I've recently deployed my application to an elastic beanstalk instance, and this is running on 64bit Amazon Linux/2.6.6.

After deploying, I'm receiving the error of 

SQLSTATE[HY000]: General error: 1 too many SQL variables

So it seems like for some reason, after changing systems, the client interacts with the SQLite driver that Archon is built on differently. I'm at a loss as to why this would work locally, but not after deployment.

Here's the code I'm using which produces the error:

$df = DataFrame::fromArray($batch_array);
$senders_emails = $df->query("SELECT a,sum(b) AS bFROM dataframe GROUP BY 1ORDER BY 2 DESC")->toArray();

Does anyone understand SQLite/Archon/Homestead vs EB well enough to help? Would greatly appreciate!

qskane
  • 481
  • 4
  • 16
Shaun Preston
  • 1

1 Answers1

0

your $batch_array have too many data.

consider this way 

$sendersEmails = [];
foreach (array_chunk($batch_array, 100) as $chunk) {
    $df = DataFrame::fromArray($chunk);
    $emails = $df->query("SELECT a,sum(b) AS bFROM dataframe GROUP BY 1ORDER BY 2 DESC")->toArray();
    $sendersEmails = array_merge($sendersEmails, $emails);
}

ATTENTION: this may lead to memory overflow

To fundamentally solve the question need to reduce the amount of $batch_array data

What is maximum query size for mysql?

qskane
  • 481
  • 4
  • 16
  • Thank you for the answer qskane, and for your function. I understand it’s due to the size of the array, however why does the same array work fine locally? It seems rather to be something to do with the machine/environment set up? – Shaun Preston Apr 26 '18 at 12:53
  • yes，generally caused by differences environment @ShaunPreston – qskane Apr 26 '18 at 13:25
  • Do you know much about what differences? Looking to learn about what’s going on behind the scenes... – Shaun Preston Apr 26 '18 at 13:47
  • I'm sorry, this is really hard to say clearly, what is the error now? @ShaunPreston – qskane Apr 26 '18 at 13:54