0

I am inserting a large amount of data. I think it is a good idea to wrap multiple INSERT statement into "batches", so I used Transaction to send to the server once every thousands INSERT statement:

        using (var reader = selectCommand.ExecuteReader())
        {
            var rowCount = 0;

            using (var transaction = mySqlConnection.BeginTransaction())
            {
                // Build the params used for insert queries
                var paramCount = reader.FieldCount;
                var queryParamBuilder = new StringBuilder();
                for (int paramNo = 0; paramNo < paramCount; paramNo++)
                {
                    queryParamBuilder.Append($"@p{paramNo},");
                }

                // Remove final comma ,
                queryParamBuilder.Remove(queryParamBuilder.Length - 1, 1);
                var queryParam = queryParamBuilder.ToString();

                while (reader.Read())
                {
                    using (var insertCommand = new MySqlCommand($"INSERT INTO {table.Name} VALUES({queryParam})", mySqlConnection))
                    {
                        insertCommand.Transaction = transaction;

                        for (int i = 0; i < paramCount; i++)
                        {
                            insertCommand.Parameters.AddWithValue($"@p{i}", reader[i]);
                        }

                        insertCommand.ExecuteNonQuery();
                    }

                    rowCount++;
                }

                transaction.Commit();
            }

I thought Transaction would help, but no, I realize data are being sent to the server every time insertCommand.ExecuteNonQuery() is called (although, correctly, the data is not there yet until Commit() is called). The reason is because, if there is a PRIMARY KEY or FOREIGN KEY error there, an exception is immediately thrown at the ExecuteNonQuery() call, not Commit() call.

Is there a way to send data to server only once? Will TransactionScope work for this case?

Luke Vo
  • 17,859
  • 21
  • 105
  • 181
  • First I recommend you to put `while (reader.Read())` inside `using (var insertCommand` – McNets Jan 10 '17 at 21:52
  • A transaction does not prevent sending data. It's allow you to discard previous command. The transaction mechanism is sever side. The client just say to the server *begin the transaction now* and *stop the transaction now*. All other operation is same as without transaction. – Kalten Jan 10 '17 at 22:00
  • @McNets My intention is to use multiple INSERT statements, but not 1 statement with multiple INSERT (because of the unknown parameters) – Luke Vo Jan 10 '17 at 22:02
  • You can reuse the same insert statement without generate a new one for every param value. It does not affect the transaction. – McNets Jan 10 '17 at 22:05
  • Wow, I didn't know they are reusable. Will try later. I am trying the adapter as suggested by Kalten. – Luke Vo Jan 10 '17 at 22:08
  • @McNets Can't reuse them, receive error "Parameter '@p0' has already been defined". – Luke Vo Jan 10 '17 at 22:30
  • @Kalten `MySqlBulkLoader` seems to support CSV or similar file only? My table is around 5GB, is there a way without creating CSV file? – Luke Vo Jan 10 '17 at 22:33
  • I also tried the Adapter. I don't know if it's my side, but the Update/Commit commands take around 12 seconds for 1000 rows, which is not really good for my need. Is there any faster way? – Luke Vo Jan 10 '17 at 22:34
  • Usually I generate an instance of my SqlCommands in class constructor, (not in a using statement), and .Add all the needed parameters without values. I suppose you get the error due `insertCommand.Parameters.AddWithValue($"@p{i}", reader[i]);` – McNets Jan 10 '17 at 22:35
  • You can try to merge many insert statements into one command. The [doc](https://dev.mysql.com/doc/refman/5.5/en/optimizing-innodb-bulk-data-loading.html) says you can enable/disable some constraint checking. That could help. – Kalten Jan 10 '17 at 22:57
  • @Kalten Your deleted link, it works like a charm. Using the CSV script + BulkLoader. 10k rows in under a second! Please mark this post as duplicate :) – Luke Vo Jan 10 '17 at 22:57
  • I put it back [SO](http://stackoverflow.com/questions/25323560/most-efficient-way-to-insert-rows-into-mysql-database). See answer with bounty too. – Kalten Jan 10 '17 at 23:03

0 Answers0