0

My C# application is significantly slower than I would like. The program accesses an Excel sheet and loops through each row on a sheet / does the following:

  1. Collects variables from that row
  2. Creates an SQL query based off those variables
  3. Executes that query
  4. then a reader goes out and puts it in its proper column on that same row.

*Note, each row has 6 different SQL queries/ final numbers that are calculated and input into the sheet, the code below is just showing the first 2 for brevity's sake. The sheet has around 300 rows, so the program is executing 300 * 6= 1,800 SQL queries each time its run. For each one of those 1,800 numbers, the program is accessing the sheet and inputting it into the sheet.

Instead of doing the excelWorksheet.get_Range and inputting the number into the Excel sheet for each number, one at a time, would it be faster to store each number into an array and then go back later and mass dump all of the numbers into the sheet, once all of the queries have been executed and the array is full?

using (SqlConnection conn = new SqlConnection())
        {
            conn.ConnectionString = "Data Source=WRDXPVSEPIRPT00;DATABASE=Epicor905;Workstation ID=SMEBPPL204TN;Trusted_Connection=true";

            try
            {
                //initiate the connection
                conn.Open();
            }
            catch(SqlException ex)
            {

                throw new ApplicationException(string.Format("You're not connected to the database, please close the program, re-connect to the network, and try again."), ex);

            }

            progressBar1.Value = 60;

            statusLabel.Text = "Retrieving account balances from database...";
            /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
            //iterate through each row and pull information
            for (int i = 2; i < lastUsedRow + 1; i++)
            {
                //string iString = Convert.ToString(i);
                statusLabel.Text = "Retrieving balances for " + i + " of " + lastUsedRow + " accounts...";



                //declare excel sheet range variables
                var CompanyVar = excelWorksheet.get_Range("A" + i, "A" + i).Text;
                var FiscalYear = excelWorksheet.get_Range("B" + i, "B" + i).Text;

                var FiscalPeriod = excelWorksheet.get_Range("C" + i, "C" + i).Text;
                var segValue1 = excelWorksheet.get_Range("F" + i, "F" + i).Text;
                var segValue2 = excelWorksheet.get_Range("G" + i, "G" + i).Text;
                var segValue3 = excelWorksheet.get_Range("H" + i, "H" + i).Text;

                int FiscalYearHelper = Convert.ToInt32(FiscalYear);
                var FiscalYearPYY = FiscalYearHelper - 1;



                //begin filling in CY YTD column                   

                string SQLCode = "SELECT SUM(DebitAmount-CreditAmount) as BalanceAmt FROM GLJrnDtl WITH (NOLOCK) WHERE" +
                             " FiscalPeriod between 1 AND " + FiscalPeriod + "AND Company =" + "'" + CompanyVar + "'" +
                             "AND FiscalYear =" + "'" + FiscalYear + "'" + "AND SegValue1 LIKE " + "'" + segValue1 + "'" +
                             "AND SegValue2 LIKE " + "'" + segValue2 + "'" + "AND SegValue3 LIKE " + "'" + segValue3 + "'";

                SqlCommand command = new SqlCommand(SQLCode, conn);


                // Create new SqlDataReader object and read data from the command.
                using (SqlDataReader reader = command.ExecuteReader())
                {
                    while (reader.Read())
                    {


                        string cyAct = reader["BalanceAmt"].ToString();
                        if (cyAct == "")
                        {
                            goto CYM;
                            //cyAct = "0.00";
                        }
                        var cyAct1 = (float)Convert.ToDouble(cyAct);
                        int one = Convert.ToInt32(excelWorksheet.get_Range("E" + i, "E" + i).Text);
                        double cyAct2 = (cyAct1 * one);
                        string cyAct3 = cyAct2.ToString("0.##");
                        excelWorksheet.get_Range("I" + i, "I" + i).Value = cyAct3;
                    }
                }
                //end filling in column


                //begin filling in CY Month column                   
                CYM:
                string SQLCodeMonth = "SELECT SUM(DebitAmount-CreditAmount) as BalanceAmt FROM GLJrnDtl WITH (NOLOCK) WHERE" +
                             " FiscalPeriod = " + FiscalPeriod + "AND Company =" + "'" + CompanyVar + "'" +
                             "AND FiscalYear =" + "'" + FiscalYear + "'" + "AND SegValue1 LIKE " + "'" + segValue1 + "'" +
                             "AND SegValue2 LIKE " + "'" + segValue2 + "'" + "AND SegValue3 LIKE " + "'" + segValue3 + "'";

                SqlCommand commandMonth = new SqlCommand(SQLCodeMonth, conn);


                // Create new SqlDataReader object and read data from the command.
                using (SqlDataReader reader = commandMonth.ExecuteReader())
                {
                    while (reader.Read())
                    {
                        string cyAct = reader["BalanceAmt"].ToString();
                        if (cyAct == "")
                        {
                            goto APY;
                            //cyAct = "0.00";
                        }
                        var cyAct1 = (float)Convert.ToDouble(cyAct);
                        int one = Convert.ToInt32(excelWorksheet.get_Range("E" + i, "E" + i).Text);
                        double cyAct2 = (cyAct1 * one);
                        string cyAct3 = cyAct2.ToString("0.##");
                        excelWorksheet.get_Range("J" + i, "J" + i).Value = cyAct3;
                    }
                }
                //end filling in column


                //begin filling in Act PY month column    
TylerH
  • 20,799
  • 66
  • 75
  • 101
Jar
  • 1,766
  • 1
  • 21
  • 27
  • 2
    Don't ever build sql queries like that: `AND Company =" + "'" + CompanyVar + "'"`. Even if your users are trusted and they will not inject bad code in excel cell which will destroy your database - company name might contain `'` symbol and your query will fail – Evk Nov 23 '17 at 16:52
  • 1
    You might find moving your SQL statement to a SP is more efficient - depending on data volume) – Paul Michaels Nov 23 '17 at 16:52
  • 6
    The Interop interface for excel is very very slow. It is better to connect to the excel worksheet using oledb. there are plenty examples on web https://www.codeproject.com/Tips/705470/Read-and-Write-Excel-Documents-Using-OLEDB – jdweng Nov 23 '17 at 16:53
  • @Evk - All users only have read access to the database. For that snipet of code that you mentioned above, how would you write it to avoid destroying the database? – Jar Nov 23 '17 at 17:18
  • 1
    @JaAustin readonly access will prevent destroying database, but will not prevent bugs. Suppose company name is "I'm the company". It contains unescaped `'` symbol which will make your query fail. Just always use parameters. – Evk Nov 23 '17 at 17:27
  • bobbytables.com for more info on this phenomenon – Caius Jard Nov 23 '17 at 17:52
  • @jdweng I ended up gaining the most from this . Due to time constraints on the project I'm unable to redo it in oledb, but ur comment made me realize I need to at least minimize my interaction with excel. So instead of going to excel each time to get the info one row at a time , I now go into excel and make a 2d array from the data at the beginning of the code, then loop through my array. I also now have the output going to another array then dumping all of the balances into excel at once, at the end. My program went from 1 minute to about 15 seconds because of this. Thx again for your help! – Jar Nov 24 '17 at 14:20
  • @jdweng I started setting up the OLEDB connection so I could use it to write to Excel but I've noticed it is very picky regarding which type / version of Excel the user is using. After researching I've noticed the forums are littered with similar issues/concerns regarding Excel version and the connection string etc. . My company is unpredictable and inconsistent in regards to driver updates, Excel updates, etc. I'm afraid some user will have an older version of Excel, or visa versa and it will throw up an error. Although interop is slow, I like the fact that it uses the .net framework – Jar Nov 24 '17 at 17:12
  • 1
    The Interop requires different version for each version of excel. Oledb there are only two versions 1) Jet :For Excel 2003 (xls) 2) Ace : For Excel 2007 or later. Jet is a window driver while ACE needs as Office license although you can done load a free version. The version is a property which is invisible inside each worksheet. – jdweng Nov 24 '17 at 18:38
  • @jdweng that makes sense, so since not everyone has the license downloaded (including me, and I don't have admin rights to download it, it would be an act of congress for ME to get it much less the entire acctg dept). So I guess with that said I could just make the excel file a .xls and use the Jet, which everyone should have , right? But The prob with that is managers are going to be using that file to change account mapping (I'm later going to sumif off of it to make reports) & I always feel sketchy when I have to work with the old version, it's kind of bad for UI. Thoughts? – Jar Nov 24 '17 at 19:52
  • @jdweng since writing this I've decreased my interactions with Excel to 2 interactions, one "read", to get the variables data for the queries, and one "write" to write the results to the sheet. I'm still not near as fast as I need to be. I have a loop counter displaying to my form, so i can see that the major deficiency is in the loop (which is now completely disassociated from Excel), so my prob must have to do with the queries. My research (in next comment) shows that the efficiency gains from Stored Procedures ,@pm_2, are minimal. I need to figure out how to reduce the number of queries. – Jar Nov 24 '17 at 23:33
  • https://stackoverflow.com/questions/15142/what-are-the-pros-and-cons-to-keeping-sql-in-stored-procs-versus-code – Jar Nov 24 '17 at 23:33
  • I don't think excel 2003 xls is going away. Too many people are using it. The only real issue is the limitations like 64,000 rows and 256 columns. Everything else is just enhancements. The jet is really the database and the excel is the front end application. So it is not excel going away it is the jet support in new versions of windows. You can create excel files on a PC with jet and no Office, just can't read the open the file with excel. – jdweng Nov 25 '17 at 02:07
  • @jdweng thanks for the info. very helpful. Since I last commented I've combined all 6 of the separate queries into one query, resulting in cutting the time in half. So now instead of opening, running, and closing 300 x 6 queries, I'm only opening running and closing 300 queries. Now I'm considering connecting via OLEDB, I think it'll save me about 3-4 seconds. – Jar Nov 25 '17 at 02:29

2 Answers2

1

THIS IS NOT A FULL ANSWER, there is not enough space to write this as a comment, do not mark this as an answer its a comment.

Please try to use objects (instances of classes or structs) it will make your programming more efficient and easy to maintain.
for example:

     private void UseObjects()
     {
         List<ExcelObjects> ListOfvarsForQuery = new List<ExcelObjects>();

         for (int i = 2; i < lastUsedRow + 1; i++)
         {
             ExcelObjects obj = new ExcelObjects();

             obj.CompanyVar = ws.Cells[i, 1];
             obj.FiscalYear = ws.Cells[i, 2];
             obj.FiscalPeriod = ws.Cells[i, 3];
             obj.segValue1 = ws.Cells[i, 4];
             obj.segValue2 = ws.Cells[i, 5];
             obj.segValue3 = ws.Cells[i, 6];
             ListOfvarsForQuery.Add(obj);
         }

         string SQLCode = // use the list of ExcelObjects and write a better query.


     }
 }

 struct ExcelObjects
 {
     public string CompanyVar;
     public string FiscalYear;
     public string FiscalPeriod;
     public string segValue1;
     public string segValue2;
     public string segValue3;
 }
Jonathan Applebaum
  • 5,738
  • 4
  • 33
  • 52
0

Instead of looping through each row and running a SQL query for each row, I found that it is faster to outer join the table I speak of in my question with another SQL table that has the account balances on it. I'm doing all of it within memory because I've been coding for 3 weeks and I don't have time to learn how to create a temp table in SQL server yet.

TylerH
  • 20,799
  • 66
  • 75
  • 101
Jar
  • 1,766
  • 1
  • 21
  • 27
  • Creating a temp table in SQL is not something you have to spend much time learning about. Simply `SELECT` your data `into #tmp_tablename` and then at the end of your query after your final `SELECT` statement, just add a line to `DROP TABLE #tmp_tablename` – TylerH Feb 28 '18 at 15:03