DataTable internal index is corrupted

Question

I am working with a .NET WinForms app in C#, running against the 3.5 .NET framework. In this app, I am setting the .Expression member of a DataColumn in a DataTable, like so:

DataColumn column = dtData.Columns["TestColumn"];
column.Expression = "some expression";

The 2nd line, where I actually set Expression, will sometimes result in the following exception:

FileName=
LineNumber=0
Source=System.Data
TargetSite=Int32 RBInsert(Int32, Int32, Int32, Int32, Boolean)
System.InvalidOperationException: DataTable internal index is corrupted: '5'.
   at System.Data.RBTree`1.RBInsert(Int32 root_id, Int32 x_id, Int32 mainTreeNodeID, Int32 position, Boolean append)
   at System.Data.RBTree`1.RBInsert(Int32 root_id, Int32 x_id, Int32 mainTreeNodeID, Int32 position, Boolean append)
   at System.Data.Index.InitRecords(IFilter filter)
   at System.Data.Index.Reset()
   at System.Data.DataTable.ResetInternalIndexes(DataColumn column)
   at System.Data.DataTable.EvaluateExpressions(DataColumn column)
   at System.Data.DataColumn.set_Expression(String value)

There is no perceptible rhyme or reason as to when the error will occur; in loading the same data set, it may work fine but then reloading it will fail, and vice versa. This leads me to think it is related to a race condition, where another write operation is occurring on the DataTable as I'm trying to modify one of its columns. However, the code relating to DataTables is not multi-threaded and runs only on the UI thread.

I have searched the web and Microsoft forums, and there is much discussion and confusion over this issue. Back when the issue was first reported in 2006, the thought was that it was an flaw in the .NET framework, and there were some supposed hotfixes released that were presumably rolled into later versions of the .NET framework. However, people have reported mixed results in applying those hotfixes, which are no longer applicable to the current framework.

Another prevailing theory is that there are operations on the DataTable which, though seemingly innocuous, are actually write operations. For example, creating a new DataView based on a DataTable is actually a write operation on the table itself, because it creates an internal index in the DataTable for later reference. These write operations are not thread-safe, so it sometimes happens that a race condition leads to an unthread-safe write coinciding with our access of the DataTable. This, in turn, causes the internal index of the DataTable to become corrupted, leading to the exception.

I have tried putting lock blocks around each DataView creation in the code, but, as I mentioned before, code utilizing the DataTable is not threaded, and the locks had no effect, in any case.

Has anyone seen this and successfully solved / worked around it?

---

No, unfortunately I can not. Loading the DataTable has already occurred by the time I get a hold of it to apply an Expression to one of its DataColumn's. I could remove the column and then re-add it using the code you suggested, but is there a particular reason why that would solve the internal index is corrupted problem?

Answer 1

I just had the same issue while importing rows, as it seems, calling DataTable.BeginLoadData before the insert fixed it for me.

Edit: As it turns out, this only fixed it on one side, now adding rows throws this exception.

Edit2: Suspending binding as suggested by Robert Rossney fixed the adding problem for me, too. I simply removed the DataSource from the DataGridView and readded it after I was done with the DataTable.

Edit3: Still not fixed...the exception keeps creeping up in all different places in my code since Thursday...this is by far the strangest and most f****ing bug I've encountered in the Framework so far (and I've seen many odd things in the 3 years I've been working with .NET 2.0, enough to warrant that not a single of my future projects will be build on it). But enough of ranting, back on topic.

I've read through the whole discussion at the Microsoft support forums and I'll give you a brief summary of it. The original bug report originates in '05.

• March '06: Bug is reported the first time, investigation starts. Throughout the course of the next year it is reported in different forms and different manifestations.
• March '07: Finally a hotfix with number KB 932491 is released (don't get your hopes up), it links against a download of an completely irrelevant looking hotfix, or at least so it seems. Throughout the next months many report that the hotfix does not work, some are reporting success.
• July '07: Last sign of live from Microsoft (with a complete useless answer), beyond this point is no further response from Microsoft. No further confirmations, no attempts on support, no requests for more information...nothing. Beyond this point there's only community related information.

No seriously, this sums it up in my opinion. I was able to extract the following information from the whole discussion:

• The DataTable is not Thread-Safe. You'll have to Lock/Synchronize it on your own if you have Multi-Threading anywhere on it.
• The corruption of the index happens somewhere before the actual exception is thrown.
• One possible corruption source is either an applied Expression or an applied Sort.
• Another possible source is the DataTable.ListChanged() event, do never modify data in this event or any event which spawns from it. This includes different Changed events from bound controls.
• There are possible issues when binding the DefaultView against a control. Always use DataTable.BeginLoadData() and DataTable.EndLoadData().
• Creation and manipulation of the DefaultView is a writing operation on the DataTable (and its Index), the Flying Spaghetti Monster knows why.

The possible source of this is most likely a race condition, either in our source code or in the code of the framework. As it seems, Microsoft is unable to fix this bug or has no intention to. Either way, check your code for race conditions, it has something to do with the DefaultView in my opinion. At some point an Insert or a manipulation of the data is corrupting the internal Index because the changes are not properly propagated through the whole DataTable.

I'll of course report back when I find further informations or additional fixes. And sorry if I get a little bit emotional here, but I've spent three days trying to pinpoint this issue, and it slowly starts to look like a good reason to get a new job.

Edit4: I was able to avoid this bug by completely removing the binding (control.DataSource = null;) and re-adding it after the loading of the data is completed. Which fuels my thought that it has something to do with the DefaultView and the events which spawn from the bound controls.

Answer 2

Personally, this particular bug has been my nemesis for 3 weeks in various fashions. I have solved it in one part of my code base and it shows up elsewhere (I believe I finally squashed it tonight). The exception info is rather unhelpful, and a way to force a reindex would have been a nice feature given the lack of MS to solve the problem.

I wouldn't look for MS's hotfix -- they have a KB article on it, then redirect you to an ASP.Net fix that is completely unrelated.

Ok - enough complaining. Let's see what has actually helped me in resolving this particular issue in the various places I've encountered it:

• Avoid using Default Views, and modifying the Default View if possible. Btw, .Net 2.0 has a number of reader/writer locks on creating views, so they are not the issue they were pre 2.0.
• Call AcceptChanges() where possible.
• Be careful about .Select(expression), since there is no reader/writer lock in this code -- and it is the only place (at least according to a person on the usenet so take it w/ a grain of salt -- however, this is very similar to your issue -- so using Mutexes may help)
• Set AllowDBNull to the column in question (questionable value, but reported on the usenet -- I've used it only in places where it makes sense)
• Make sure that you are not setting null (C#)/Nothing (VB) to a DataRow field. Use DBNull.Value instead of null. In your case you may wish to check that the field is not null, the expression syntax does supports the IsNull(val, alt_val) operator.

• This has probably helped me the most (absurd as it sounds): If a value is not changing, don't assign it. So in your case use this instead of your outright assignment:

  if (column.Expression != "some expression") column.Expression = "some expression";

(I removed the square brackets, not sure why they were there).

Edit (5/16/12): Just ran into this issue repeatedly (with an UltraGrid/UltraWinGrid). Used the advice of removing the sort on the DataView, and then added a sorted column which matched the DataView sort, and this resolved the issue.

Answer 3

Just a note for those trying to see how this bug can be reproduced. I have some code which will fairly often produce that error. It does locking around concurrent read/writes, but the call to DataView.FindRows is done outside of that locking. The OP pointed out that creating a data view was a hidden write operation, is querying it one as well?

//based off of code at http://support.microsoft.com/kb/932491
using System.Data;
using System.Collections.Concurrent;
using System.Threading.Tasks;
using System;
public class GenerateSomeDataTableErrors
{   
    public static void Main()
    {
        DataTable Table = new DataTable("Employee");
        Table.Columns.Add("Id", typeof(int));
        Table.Columns.Add("Name", typeof(string));
        Table.PrimaryKey = new DataColumn[] { Table.Columns["Id"] };

        DataSet Employees = new DataSet();
        Employees.Tables.Add(Table);

        DataRow ManagerB = Table.NewRow();
        ManagerB["ID"] = 392;
        ManagerB["Name"] = "somename";
        Table.Rows.Add(ManagerB);

        DataRow ManagerA = Table.NewRow();
        ManagerA["ID"] = 394;
        ManagerA["Name"] = "somename";
        Table.Rows.Add(ManagerA);

        Employees.AcceptChanges();

        object locker = new object();

        //key = exception string, value = count of exceptions with same text
        ConcurrentDictionary<string, int> exceptions = new ConcurrentDictionary<string, int>();

        DataView employeeNameView = new DataView(Table, string.Empty, "Name", DataViewRowState.CurrentRows);

        Parallel.For(0, 100000, (i, s) =>
        {
            try
            {
                #region do modifications to the table, in a thread-safe fashion
                lock (locker)
                {
                    var row = Table.Rows.Find(392);

                    if (row != null) //it's there, delete it
                    {
                        row.Delete();
                        Employees.AcceptChanges();
                    }
                    else //it's not there, add it
                    {
                        var newRow = Table.NewRow();
                        newRow["ID"] = 392;
                        newRow["Name"] = "somename";
                        Table.Rows.Add(newRow);
                        Employees.AcceptChanges();
                    }
                }
                #endregion

                //Apparently this is the dangerous part, finding rows 
                // without locking on the same object the modification work is using.
                //lock(locker)
                employeeNameView.FindRows("somename");
            }
            catch (Exception e)
            {
                string estring = e.ToString();
                exceptions.TryAdd(estring, 0);
                lock (exceptions)
                { exceptions[estring] += 1; }
            }
        });

        foreach (var entry in exceptions)
        {
            Console.WriteLine("==============The following occurred " + entry.Value + " times");
            Console.WriteLine(entry.Key);
        }
    }//Main
}//class

If you run it as-is you could get output like this (output differs somewhat each time you run it):

==============The following occurred 2 times
System.InvalidOperationException: DataTable internal index is corrupted: '13'.
   at System.Data.RBTree`1.GetNodeByIndex(Int32 userIndex)
   at System.Data.DataView.GetRow(Int32 index)
   at System.Data.DataView.GetDataRowViewFromRange(Range range)
   at System.Data.DataView.FindRowsByKey(Object[] key)
   at GenerateSomeDataTableErrors.<>c__DisplayClass9.<Main>b__8(Int32 i, ParallelLoopState s) in Program.cs:line 110
==============The following occurred 3 times
System.IndexOutOfRangeException: Index 1 is either negative or above rows count.
   at System.Data.DataView.GetRow(Int32 index)
   at System.Data.DataView.GetDataRowViewFromRange(Range range)
   at System.Data.DataView.FindRowsByKey(Object[] key)
   at GenerateSomeDataTableErrors.<>c__DisplayClass9.<Main>b__8(Int32 i, ParallelLoopState s) in line 110
==============The following occurred 1 times
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Data.DataView.GetRow(Int32 index)
   at System.Data.DataView.GetDataRowViewFromRange(Range range)
   at System.Data.DataView.FindRowsByKey(Object[] key)
   at GenerateSomeDataTableErrors.<>c__DisplayClass9.<Main>b__8(Int32 i, ParallelLoopState s) in Program.cs:line 110
Press any key to continue . . .

and if you do put the lock on the FindRows call, no exceptions.

Answer 4

You mention "not threadsafe". Are you manipulating the object from different threads? If so then that might very well be the reason for the corruption.

Answer 5

My understanding, from long and painful haggling over this problem, is that it's an artifact of non-thread-safe write operations, which usually you didn't know you were making.

In my case, the culprit seemed to be the BindingSource. I found that I needed to suspend binding, perform whatever operation I was trying, and then resume binding when I was done, and the problem went away. This was 18 months ago, so I'm not clear on the details anymore, but I remember getting the impression that the BindingSource is doing some kind of operation on its own thread. (This makes less sense to me now than it did at the time.)

Another potential source of trouble is the DataTable's RowChanging event. If you do something that modifies the table in that event handler, expect bad things.

Answer 6

I resolved my datatable-internal-index error this way:

changed CellEndEdit to CellBeginEdit event. Also... avoid using NULLs unnecessarily:

Private Sub User_role_groupDataGridView_CellBeginEdit(sender As Object, e As DataGridViewCellCancelEventArgs) Handles User_role_groupDataGridView.CellBeginEdit
    Try 
        If Not Me.User_role_groupDataGridView.Rows(e.RowIndex).IsNewRow Then Me.User_role_groupDataGridView.Rows(e.RowIndex).Cells("last_modified_user_group_role").Value = Now
    Catch ex As Exception
        Me.displayUserMessage(ex.ToString, Me.Text, True)
    End Try
End Sub

Answer 7

I've encountered the same issue and this is what fixed it for me: Stack Overflow - internal index is corrupted.

If you are using threads with a dataset, that error will occur.

In my case, I was trying to create a new row for a dataset within a method that was running in threads.

One way was to use SyncLock around the method that creates the row or another way (and probably even better) was to create the rows outside of threads.

Basically my code looks something like this:

    Dim elements As New List(Of element)
    Dim dataRows As New List(Of MyDataSet.Row)

    For i As Integer = 0 To elements.Count - 1
        dataRows.Add(Me.Ds.Elements.NewElementsRow)
    Next

    Parallel.For(0, elements.Count, Sub(i As Integer)
                                        Me.CreateElementRow(elements(i), dataRows(i))
                                    End Sub)

In the CreateElementRow method I'm doing a lot of calculations in threads.

Hope this helps.

Answer 8

I had the same issue (table index corrupted with 5) on adding rows programaticly to a dataset which is bound to the datagridview. I didn't take into account that there was an eventhandler on the AddRow event of the datagridview, which does some initialization in case the user starts the new row by UI. In exceptions stack trace nothing was seen of it. By disabling the event I could solve this fast. I came to it only by reading some comments here in deep. 2hours not too much for issues like that :-), I think. You can find this by setting breakpoint in every eventhandler assigned to datgridview which is linked to the dataset.

Answer 9

Same prob over here, and tried a different approach. I'm not using the datatable for any screen related stuff (e.g. binding); I'm just creating DataRow objects (in multiple threads) and adding them to the table.

I've tried using lock(), and also tried centralizing the addition of the rows into a singleton, thinking this would help. It didn't. For reference, here's the singleton I used. Maybe someone else will be able to build on this and figure something out?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data;

namespace EntityToDataSet
{
   public class RowAdder
   {
      #region Data
      private readonly object mLockObject = new object();
      private static RowAdder mInstance;

      public static RowAdder Instance
      {
         get
         {
            if (mInstance == null)
            {
               mInstance = new RowAdder();
            }
            return mInstance;
         }
      }

      object mSync;
      #endregion

      #region Constructor
      private RowAdder()
      {
      }
      #endregion

      public void Add(DataTable table, DataRow row)
      {
         lock (mLockObject)
         {
            table.Rows.Add(row);
         }
      }
   }
}

Answer 10

How about trying the idea of applying a mutex as described here to induce sleep in the thread in such conditions?

Answer 11

This is how I fixed my Internal index is corrupted problem:

System.Data.DataTable dtNew = new DataTable();
for (int iCol = 0; iCol < dtOriginalData.Columns.Count; iCol++)
{
    dtNew.Columns.Add(dtOriginalData.Columns[iCol].ColumnName, dtOriginalData.Columns[iCol].DataType);
}
for (int iCopyIndex = 0; iCopyIndex < item.Data.Rows.Count; iCopyIndex++)
{
    dtNew.Rows.Add(dtOriginalData.Rows[iCopyIndex].ItemArray);
    //dtNew.ImportRow(dtOriginalData.Rows[iCopyIndex]); 
}
dtOriginalData = dtNew; 

Enjoy, Andrew M

Answer 12

May be you are using same datatable in mutiple process at same time.. I just resolved this issues using SYNCLOCK...

Try this..

SyncLock your datatable

'''' ----your datatable process

End SyncLock

Answer 13

In my case the Framework version is 2.0. The source of a problem was in DataView ListChanged event. The code below initializes the new row with some default values.

private void dataView_ListChanged(object sender, ListChangedEventArgs e)
{
    if (e.ListChangedType == ListChangedType.ItemAdded)
    {
        DataView v = (DataView)sender;
        DataRowView drv = v[e.NewIndex];

        // This "if" works fine
        if (drv["Foo"] == DBNull.Value)
        {
            drv["Foo"] = GetFooDefault();
        }

        // This "if" brakes the internal index     
        if (drv["Bar"] == DBNull.Value && drv["Buz"] != DBNull.Value)
        {
            drv["Bar"] = drv["Buz"];
        }
    }
}

After some investigation it became clear that ItemAdded event is called at least twice per row. First time when the UI creates new line for entering data and second time, well I'm not sure, but looks like when the DataRowView is added to a DataView.

The first "if" works only when the ItemAdded is called first time. On second call "Foo" column is already populated and left as is.

However the "Bar" column defaulting code can be executed on both calls. Actually in my case it was executed only on second ItemAdded event, when the user had a chance to fill in the data for "Buz" column (initially "Buz" has DBNull value).

So here is recommendations based on my findings:

• The data in ListChanged event can be changed only when e.ListChangedType == ListChangedType.ItemAdded.
• Prior to setting the column value the check should be performed to ensure that this is the first ItemAdded event (e.g. if the value cannot be null on second call, check if it is DBNull.Value etc.)

Answer 14

Here is what seems to have worked for my colleague Karen and I. We were getting this error in a DataGridView, but only when entering data into one particular column.

It turns out that I had changed the order of columns in the grid, not knowing that there was code in the DataGridView.CellValidated sub that nulls out the value in that one particular column causing the problem.

That code referred to a specific column number. So when the original DataGridView column 3 was moved and became column 1, but the DataGridView.CellValidated code still referred to column 3, the error occurred. Changing our code so that it referred to the correct e.ColumnIndex seems to have fixed our problem.

(It was not easy to figure out to change this one number in our code. I hope this fix holds.)

Answer 15

can't you just use:

dtData.Columns.Add("TestColumn", typeof(Decimal), "Price * Quantity");

Answer 16

I had the same problem using Threads. What I did was create a delegate which is called when I need to merge the table.

internal delegate void MergeData (DataTable dataTable1, DataTable dataTable2);

internal static void MergeDataTable (DataTable dataTable1, DataTable dataTable2)
{
    dataTable1.Merge (dataTable2, true);
}

Then during execution I call the delegate and the error does not occur.

Delegates.MergeData mergeData = new Delegates.MergeData (Delegates.MergeDataTable);

object [] paramsMerge = {dataTable1, dataTable2};

this.Invoke (mergeData, paramsMerge);

Answer 17

Had the same with .NET 4.5.2 win forms app. In my case the reason appeared to be multiple controls bound to single BindingSource column. I am aware that binding many controls to one value is asking for trouble but I it has been the lesser evil for me due to pretty complex layout.

It appeared that changing the value from one side raised multiple OnChange events trying to perform the same operation against the BindingSource which resulted with the error in question.

I have tried suspending binding but with no success. I have implemented flags preventing the code from being executed many times on parallel and things are better.

Answer 18

Common DataTable doesn't supports concurrent operations. Use sqlite or similar orm database solution.

Currently workaround is created all specified rows and then fill in parallel.

BAD:

Parallel.ForEach(needles, (needle, state, index) => {
    DataTable.Rows.Add(needle);
});

GOOD:

foreach(var needle in needles) DataTable.Rows.Add();
Parallel.ForEach(needles, (needle, state, index) => {
    DataTable.Rows[(int)index].ItemArray(0) = needle;
});

Answer 19

The same happened to me too. Winforms, .NET 3.5, unexpectectly got this error trying to set one of the columns in typed row. Code was rather old and worked for a long time so it was kind of unpleasant surprise...

I needed to set new SortNo's in typed table TadaTable in dataset TadaSet.

What helped me, you can also try this:

int i = 0;
foreach (TadaSet.TadaTableRow row in source)
{
     row.BeginEdit(); //kinda magical operation but it helped :)
     // Also you can make EndEdit() for each row later if you need...
     short newNo = i++;
     if (newNo != row.SortNo) row.SortNo = newNo; //here was the crash
}