1

I know this question exists, because it's mine and I put up 500 bounty points on it:

Exporting C# report to Excel when there are more than 5K lines

The answer got me over the hump (to some degree) but we're sort of at the point where we just accept that abnormally large datasets just can't be exported via our ASP front end, so we ship those requests off to our SQL Server DBs, who then run the appropriate stored procedures and copy/paste to Excel spreadsheets.

My question here is; can someone definitively answer whether or not it's absolutely impossible to export a large dataset to an Excel spreadsheet via a ASP front end? Once a particular report hits about 8K records or something, it just can't seem to be done. I'm just trying to determine whether any other potential tweak can be made, or if that much data is just more than ASP can handle?

Community
  • 1
  • 1
Johnny Bones
  • 8,786
  • 7
  • 52
  • 117
  • If you can narrow down the question, for example, 'how to export C# DataTable with more than 5K records to Excel worksheet' (2007 and up), then I probably have an answer. Rgds, – Alexander Bell Jun 30 '14 at 15:47
  • That pretty much sounds like exactly what I asked, although technically it's a SQL Server datatable. – Johnny Bones Jun 30 '14 at 15:51
  • This is more a limitation of Excel than ASP.NET. See http://superuser.com/questions/366468/what-is-the-maximum-allowed-rows-in-a-microsoft-excel-xls-or-xlsx – David Jun 30 '14 at 15:55
  • Where your problem is? In exporting? In downloading? – Adriano Repetti Jun 30 '14 at 15:56
  • The problem is in exporting. Anything more than 7 or 8K records and it just does nothing. – Johnny Bones Jun 30 '14 at 15:58
  • thought about using webapi and JSON, then using javascript you can easily convert http://stackoverflow.com/questions/4130849/convert-json-format-to-csv-format-for-ms-excel, pretty sure that would get round any asp.net limits. – fuzzybear Jun 30 '14 at 15:59
  • A reach but check this out. http://forums.asp.net/t/1045565.aspx?inserting+data+in+excel+limited+to+255+char+ At a certain size it does not sample the whole data set. – paparazzo Jun 30 '14 at 16:09
  • @Johny Bones : I have deleted my solution as some folks seemingly don't like it a lot. Hopefully, they will be able to offer you a better one. Kind regards, – Alexander Bell Jun 30 '14 at 20:24

2 Answers2

0

Well... since I've streamed gigabytes of data directly from ASP.NET, I'm pretty sure you're doing something wrong. Try to isolate the problem first - is it in putting the data into the session, is it request / response limits, is it request timeouts? Figure out where the problem is, and then go ahead and solve it! :)

In general terms, there's no reason why you should put the data in a DataSet first. Instead, use a SqlDataReader and write the data to output in chunks. This way you'll avoid having the whole data set in memory; the same way, you can just directly write to the output stream, without buffering the generated HTML in memory. Why do you keep data in Session? Wouldn't it be better to just hold the parameters necessary to retrieve it from the DB as needed, using the DataReader?

If you're having trouble with timeouts, periodical Flushes help. This also helps reduce the memory footprint on the ASP.NET side.

Saving the output data to a file on the server first also helps, and it allows you to wire up partial file downloads too - just make sure you actually have enough space on the drive.

EDIT:

Ok, so you've got an SqlCommand. Instead of using it in a SqlDataAdapter, you can do something like this (cmd being your SqlCommand instance):

HtmlTextWriter wr = new HtmlTextWriter(Response.Output);

using (var rdr = cmd.ExecuteReader())
{
    int index = 0;

    wr.WriteBeginTag("table");
    wr.WriteLine("<tr><td>Column 1</td><td>Column 2</td></tr>");

    while (rdr.Read())
    {
        wr.WriteBeginTag("tr");

        wr.WriteBeginTag("td");
        wr.Write(rdr["Column1"]);
        wr.WriteEndTag("td");

        wr.WriteBeginTag("td");
        wr.Write(rdr["Column2"]);
        wr.WriteEndTag("td");

        wr.WriteEndTag("tr");

        if (index++ % 1000 == 0) Response.Flush();
    }

    wr.WriteEndTag("table");
}

I have not tested it, so it might need some tweaking, but the idea should be pretty obvious.

Luaan
  • 62,244
  • 7
  • 97
  • 116
0

It is possible to do this as I have actually just finished some code specifically to do this as part of a reporting project that I am working on where we have in-excess of 20K records that need to be pulled back and exported into excel.

I will pull out the code and stick it on github for you to look at.

I am actually using NPOI's excel processing package and then using my custom code I am able to process any List of classes dynamically into a dataset and then dump it into the worksheets.

I need to tidy up the code for you but I should have something ready for you this evening.

This code will work for both desktop and web apps.

To give you an idea my code has been able to process a dataset of over 30K relatively quickly. I have to resolve an issue with datasets over the limit of 65536 records first before it is ready for you.

The nice thing with this solution means it doesn't rely on excel being installed on the machine hosting the solution.

EDIT I have loaded a project onto github here:

https://github.com/JellyMaster/ExcelHelper

but here is the main bit that does all the excel processing:

    public static MemoryStream CreateExcelSheet(DataSet dataToProcess)
    {
        MemoryStream stream = new MemoryStream();

        if (dataToProcess != null)
        {
            var excelworkbook = new HSSFWorkbook();

            foreach (DataTable table in dataToProcess.Tables)
            {
                var worksheet = excelworkbook.CreateSheet();

                var headerRow = worksheet.CreateRow(0);

                foreach (DataColumn column in table.Columns)
                {
                    headerRow.CreateCell(table.Columns.IndexOf(column)).SetCellValue(column.ColumnName);

                }

                //freeze top panel. 
                worksheet.CreateFreezePane(0, 1, 0, 1);


                int rowNumber = 1;

                foreach (DataRow row in table.Rows)
                {
                    var sheetRow = worksheet.CreateRow(rowNumber++);

                    foreach (DataColumn column in table.Columns)
                    {
                        sheetRow.CreateCell(table.Columns.IndexOf(column)).SetCellValue(row[column].ToString());
                    }


                }

            }

            excelworkbook.Write(stream);
        }



        return stream;
    }

    public static DataSet CreateDataSetFromExcel(Stream streamToProcess, string fileExtentison = "xlsx")
    {
        DataSet model = new DataSet();




        if (streamToProcess != null)
        {

            if (fileExtentison == "xlsx")
            {
                XSSFWorkbook workbook = new XSSFWorkbook(streamToProcess);

                model = ProcessXLSX(workbook);

            }
            else
            {
                HSSFWorkbook workbook = new HSSFWorkbook(streamToProcess);

                model = ProcessXLSX(workbook);
            }





        }

        return model;

    }

    private static DataSet ProcessXLSX(HSSFWorkbook workbook)
    {

        DataSet model = new DataSet();
        for (int index = 0; index < workbook.NumberOfSheets; index++)
        {
            ISheet sheet = workbook.GetSheetAt(0);

            if (sheet != null)
            {
                DataTable table = GenerateTableData(sheet);

                model.Tables.Add(table);

            }



        }



        return model;

    }

    private static DataTable GenerateTableData(ISheet sheet)
    {

        DataTable table = new DataTable(sheet.SheetName);

        for (int rowIndex = 0; rowIndex <= sheet.LastRowNum; rowIndex++)
        {
            //we will assume the first row are the column names 
            IRow row = sheet.GetRow(rowIndex);

            //a completely empty row of data so break out of the process.
            if (row == null)
            {

                break;
            }

            if (rowIndex == 0)
            {


                for (int cellIndex = 0; cellIndex < row.LastCellNum; cellIndex++)
                {
                    string value = row.GetCell(cellIndex).ToString();

                    if (string.IsNullOrEmpty(value))
                    {
                        break;
                    }
                    else
                    {
                        table.Columns.Add(new DataColumn(value));
                    }

                }
            }
            else
            {
                //get the data and add to the collection 

                //now we know the number of columns to iterate through lets get the data and fill up the table. 

                DataRow datarow = table.NewRow();

                object[] objectArray = new object[table.Columns.Count];

                for (int columnIndex = 0; columnIndex < table.Columns.Count; columnIndex++)
                {
                    try
                    {
                        ICell cell = row.GetCell(columnIndex);

                        if (cell != null)
                        {
                            objectArray[columnIndex] = cell.ToString();
                        }
                        else
                        {
                            objectArray[columnIndex] = string.Empty;
                        }



                    }
                    catch (Exception error)
                    {
                        Debug.WriteLine(error.Message);
                        Debug.WriteLine("Column Index" + columnIndex);
                        Debug.WriteLine("Row Index" + row.RowNum);
                    }
                }


                datarow.ItemArray = objectArray;
                table.Rows.Add(datarow);

            }



        }

        return table;
    }

    private static DataSet ProcessXLSX(XSSFWorkbook workbook)
    {
        DataSet model = new DataSet();
        for (int index = 0; index < workbook.NumberOfSheets; index++)
        {
            ISheet sheet = workbook.GetSheetAt(index);

            if (sheet != null)
            {
                DataTable table = GenerateTableData(sheet);

                model.Tables.Add(table);

            }



        }



        return model;
    }




}

This does require the NPOI nuget package to be installed in your project.

Any questions give me a shout. The github project does a bit more but this is enough to get you going hopefully.

David Shorthose
  • 4,489
  • 2
  • 13
  • 12