1

I am generating a SQL insert statement within a PHP for loop.

The SQL string generated is a large number of individual SQL statements like this:

INSERT INTO tbl VALUES(1,2,3);
INSERT INTO tbl VALUES(4,5,6);
INSERT INTO tbl VALUES(7,8,9);

etc...

Then I execute with:

$InsertResult = sqlsrv_query($conn, $InsertSQL);

The problem is that only the first 312 statements get executed instead of the full 2082 lines (only 312 rows are inserted into the table).

When I output the $InsertSQL variable to the JavaScript console and then execute it manually in SSMS it works perfectly and inserts all 2082 lines. Only when I run the $InsertSQL variable through sqlsrv_query does it not go to completion.

I also don't get any errors and the query result comes back true as tested in this line:

if(!$InsertResult) die('Problem with Insert query: ' . $InsertSQL);

When I searched for a solution to this problem I saw that (although it's not mentioned in the PHP manual site) sqlsrv_query apparently has a string character limit on the $SQL variable (around 65k characters).

See the other StackOverflow article here: length restriction for the sql variable on sqlsrv_query?

I figured this was the problem and so created a shorter version of the string (by only adding in the column values that I actually wanted to import). This much shorter version however, still only Inserts the first 312 lines! So now it seems this is NOT related to the max string length. In fact, if it was, I should only get around 250 lines (after 250 statements I'm at about 65k characters).

I can also execute each insert statement individually but of course this takes much longer. In my testing, it takes 90s or so to do it this way where as running the combined statement manually in SMSS takes only around 40s.

Note that I've also looked into SQL Server's Bulk Insert however I won't be able to send the file to the machine where SQL Server is installed (SQL Server and Web servers are on separate computers). From my understanding this eliminates this possibility.

Any help is much appreciated as I can't even figure out what it is that is limiting me, never mind fix it and I'd hate to have to execute one line at a time.

Zhorov
  • 28,486
  • 6
  • 27
  • 52
CShore
  • 65
  • 8
  • Is there any reason why you aren't doing this as one insert? It would be way more performant – John Conde Jun 04 '20 at 01:36
  • Yes, there is a 1000 row limit if done that way. My fallback solution is to go back to that but every 999 rows execute the Insert statement then start again until complete. Bit messy though and still requires multiple insert statements in the end (although much less) – CShore Jun 04 '20 at 01:39
  • 1
    So do three queries.Still way faster – John Conde Jun 04 '20 at 01:40
  • I edited my comment and addressed that possibility. – CShore Jun 04 '20 at 01:41
  • Note that I won't typically know ahead of time how many rows there are. would need to build that in. – CShore Jun 04 '20 at 01:43
  • Maybe there's a 1000 row limit in PHP, I've not observed such a thing elsewhere. You don't need to send a file to the server to use BULK INSERT, you can use the [bcp.exe utility](https://learn.microsoft.com/en-us/sql/tools/bcp-utility) from your local command line. – AlwaysLearning Jun 04 '20 at 02:46
  • 1
    @AlwaysLearning SQL Server has a 1000 row limit on INSERT INTO ...VALUES() statements. – CShore Jun 04 '20 at 03:00
  • @JohnConde I went ahead and implemented your suggestion. Every 1000 rows I submit and then start over. Ended up with a result in the middle at 60s. Better than 90s for longest approach of one line at a time. Slower than SMSS executing the full multi statement version of 40s. If it wasn't for sqlsrv's issue could still do 50% better though. – CShore Jun 04 '20 at 03:03
  • @JohnConde Actually it's now working great! I'm down to 10s! I'm using Azure for SQL Server and not on a high paying tier. It seems the more I run the update, the faster it's getting... first went to 40s, 20s then10s... In any case, it works! Thanks. – CShore Jun 04 '20 at 03:21

1 Answers1

2

Explanations:

There is a known issue with this driver, posted on GitHub, about executing large SQL statements. One part of the provided solution are the following explanations:

Seems like when executing a large batch of SQL statements, Microsoft SQL Server may stop processing the batch before all statements in the batch are executed. When processing the results of a batch, SQL Server fills the output buffer of the connection with the result sets that are created by the batch. These result sets must be processed by the client application. If you are executing a large batch with multiple result sets, SQL Server fills that output buffer until it hits an internal limit and cannot continue to process more result sets. At that point, control returns to the client. This behavior is by design. Client app should flush all the pending result sets. As soon as all pending result sets are consumed by the client, SQL Server completes executing the batch. Client app can call sqlsrv_next_result() until it returns NULL.

So, I don't think that there is a limit for the SQL statement length, only the size of a PHP string variable ($InsertSQL in your case) is limited to the maximum allowed PHP memory limit. The actual reason for this unexpected behaviour is the fact, that with SET NOCOUNT OFF (this is by default) and a large number of single INSERT statements, the SQL Server returns the count of the affected rows as a result set (e.g. (1 row affected)).

Solution:

I'm able to reprodiuce this issue (using SQL Server 2012, PHP 7.1.12 and PHP Driver for SQL Server 4.3.0+9904) and you have the following options to solve this problem:

  • Flush the pending result sets using sqlsrv_next_result().
  • Execute SET NOCOUNT ON as first line in your complex T-SQL statement to stop SQL Server to return the count of the affected rows as a resultset.
  • Use parameterized statement using sqlsrv_prepare()\sqlsrv_execute()

Table:

CREATE TABLE MyTable (
    Column1 int,
    Column2 int,
    Column3 int
)

One complex statement (using sqlsrv_query() and sqlsrv_next_result()):

<?php 

// Connection info
$server = 'server\instance';
$database = 'database';
$username = 'username';
$password = 'password';
$cinfo = array(
    "Database" => $database,
    "UID" => $username,
    "PWD" => $password
);

// Statement with sqlsrv_query
$sql = "";
for ($i = 1; $i <= 1000; $i++) {
    $sql .= "INSERT INTO MyTable (Column1, Column2, Column3) VALUES (".$i.", 0, 0);";
}
$stmt = sqlsrv_query($con, $sql);
if ($stmt === false) {
    echo "Error (sqlsrv_query): ".print_r(sqlsrv_errors(), true);
    exit;
}

// Clean the buffer
while (sqlsrv_next_result($stmt) != null){};

// End
sqlsrv_free_stmt($stmt);
sqlsrv_close($con);
echo "OK";
?>

One complex statement (using sqlsrv_query() and SET NOCOUNT ON):

<?php 

// Connection info
$server = 'server\instance';
$database = 'database';
$username = 'username';
$password = 'password';
$cinfo = array(
    "Database" => $database,
    "UID" => $username,
    "PWD" => $password
);

// Connection
$con = sqlsrv_connect($server, $cinfo);
if ($con === false) {
    echo "Error (sqlsrv_connect): ".print_r(sqlsrv_errors(), true);
    exit;
}

// Statement with sqlsrv_query
$sql = "SET NOCOUNT ON;";
for ($i = 1; $i <= 1000; $i++) {
    $sql .= "INSERT INTO MyTable (Column1, Column2, Column3) VALUES (".$i.", 0, 0);";
}
$stmt = sqlsrv_query($con, $sql);
if ($stmt === false) {
    echo "Error (sqlsrv_query): ".print_r(sqlsrv_errors(), true);
    exit;
}

// End
sqlsrv_free_stmt($stmt);
sqlsrv_close($con);
echo "OK";
?>

Parameterized statement (using sqlsrv_prepare() and sqlsrv_execute()):

<?php 

// Connection info
$server = 'server\instance';
$database = 'database';
$username = 'username';
$password = 'password';
$cinfo = array(
    "Database" => $database,
    "UID" => $username,
    "PWD" => $password
);

// Connection
$con = sqlsrv_connect($server, $cinfo);
if ($con === false) {
    echo "Error (sqlsrv_connect): ".print_r(sqlsrv_errors(), true);
    exit;
}

$sql = "INSERT INTO MyTable (Column1, Column2, Column3) VALUES (?, ?, ?);";
$value1 = 0;  
$value2 = 0;  
$value3 = 0;  
$params = array(&$value1, &$value2, &$value3);
$stmt = sqlsrv_prepare($con, $sql, $params);
if ($stmt === false ) {
    echo "Error (sqlsrv_prepare): ".print_r(sqlsrv_errors(), true);
    exit;
}
for ($i = 1; $i <= 1000; $i++) {
    $value1 = $i;  
    $value2 = 0;  
    $value3 = 0;  
    $result = sqlsrv_execute($stmt);
    if ($result === false) {
        echo "Error (sqlsrv_execute): ".print_r(sqlsrv_errors(), true);
        exit;
    }
}

// End
sqlsrv_free_stmt($stmt);
sqlsrv_close($con);
echo "OK";
?>
Zhorov
  • 28,486
  • 6
  • 27
  • 52
  • 1
    The app is being tested right now with the multi batch approach and is working OK. Once testing is done I will certainly give this a try. Looks like this is the issue though. I didn't find that article, thanks for the link and for all 3 detailed solutions! Great to know it wasn't just me! SUPER helpful! Thanks again! – CShore Jun 09 '20 at 15:49