Mysterious behavior of Dictionary

Question

I'm working on a huge system based on Asp.net MVC 3.0 and working on Mono-2.10.8 (Windows 7).

Everything was fine until a moment couple of days ago.

Inside my API I have several utility classes using dictionaries. For example, like this one:

public static class KeyUtility  
{
  static KeyUtility() {
    Alphabet = new[] {
      'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 
      'J', 'K', 'L', 'M', 'N', 'P', 'R', 'S', 
      'T', 'U', 'V', 'X', 'Y', 'Z', '0', '1', 
      '2', '3', '4', '5', '6', '7', '8', '9'
    };

    ReverseAlphabet = Alphabet
      .Select((c, i) => new { Char = c, Value = i })
      .ToDictionary(k => k.Char, v => (byte) v.Value);
  }

  internal static char[] Alphabet;      
  private static IDictionary<char, byte> ReverseAlphabet;

  public static string ToKey(byte[] key, int groupSize)
  {
    //Accessing Alphabet to generate string key from bytes
  }

  public static byte[] FromKey(string key)
  {
    //Accessing ReverseAlphabet to get bytes from string key
  }
}

And randomly I get exceptions like this:

System.IndexOutOfRangeException: Array index is out of range.
at System.Collections.Generic.Dictionary`2<char, byte>.TryGetValue (char,byte&) <0x000a1>
at MyAPI.KeyUtility.FromKey (string) <0x0009a>
at MyApp.Controllers.AboutController.Index () <0x002df>
at (wrapper dynamic-method) object.lambda_method (System.Runtime.CompilerServices.Closure,System.Web.Mvc.ControllerBase,object[]) <0x0002f>
at System.Web.Mvc.ActionMethodDispatcher.Execute (System.Web.Mvc.ControllerBase,object[]) <0x0001b>
at System.Web.Mvc.ReflectedActionDescriptor.Execute (System.Web.Mvc.ControllerContext,System.Collections.Generic.IDictionary`2<string, object>) <0x000ff>
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod (System.Web.Mvc.ControllerContext,System.Web.Mvc.ActionDescriptor,System.Collections.Generic.IDictionary`2<string, object>) <0x00019>
at System.Web.Mvc.ControllerActionInvoker/<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12 () <0x00066>
at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter (System.Web.Mvc.IActionFilter,System.Web.Mvc.ActionExecutingContext,System.Func`1<System.Web.Mvc.ActionExecutedContext>) <0x000b8>

Most of the time everything is fine and KeyUtility works correct, but on rare occasions I get such an exception. Despite it looks like thread safety issue, the dictionary ReverseAlphabet is always accessed for reading only and never for writing. Once it's created in static constructor it's only accessed with TryGetValue. As I understand from MSDN article it should be thread safe in this case. Moreover, I haven't seen this problem before.

What should I look at? I even can't create a reproduction as I have completely no idea what is wrong.

Mono-2.10.8 and older versions proved to be stable with dictionaries. I've been using it for a couple of years intensively and have never seen this sort of exception before.

How to fix this?

UPD:

I've remembered that near the time of the begining of troubles what i've done is statically linked mono with my executable (i'm embedding mono into my application). I simply downloaded sources of mono. Compilled it without any changes except i set up libmono's output to static library. I've also linked with libeay32 and sqlite3. All multithread (MT). Maybe this change could affect an application? Unfortunately i can't check this under standalone mono. Before this i was linking all libraries dynamically and everything was fine.

UPD2: Here is the link to the complete sources: http://pastebin.com/RU4RNCki

Are you sure it is **only** used in a read way? ***absolutely*** sure it is never updated in any way? — Marc Gravell, May 23 '12 at 09:54
I've looked at `TryGetValue` using `dotPeek 1.0`, and from that, I can only suggest that items must be deleted (taking into account your putting items in the dictionary at the start) from the internal array for this exception to be raised. Either that or Mono's interpretation of a generic dictionary might have a flaw? — Jason Evans, May 23 '12 at 09:58
I am looking at the sources and Dictionary.TryGetValue has not changed between Mono 2.8 and 2.11.1, so we can rule out a change in the implementation. Reading the implementation code also indicates that the only possibility of an IndexOutOfRangeException here is if the dictionary changes while it is being accessed. I recommend you look for hidden places where this may be happening. — Sander, May 23 '12 at 10:07
@MarcGravell yes, i'm absolutely sure it is. There are only 2 reference to it: in static ctor and in From key method calling TryGetValue. — ILya, May 23 '12 at 10:07
@ThomasLevesque unfortunately yes. That's why i'm loosing my mind trying to find what causes an exception — ILya, May 23 '12 at 10:09
@JasonEvans it could be. i've also reflected it, but i have strong doubts because i've never faced this before. And googling on the topic had no effect for me — ILya, May 23 '12 at 10:11
Maybe some other part of this solution is doing something sneaky like using reflection to change this dictionary without ever referencing it? I suppose you could make sure by creating an IDictionary wrapper for this that throws exceptions on everything except TryGetValue and Add. — Sander, May 23 '12 at 10:13
@Sander From the one hand, i'm sure that mono's implementation of dictionary is ok, from the other, i'm absolutely sure in my code. The class is only 150 lines and only one call of TryGetValue, nothing more. It's also not accessed via reflection (for sure i've done whole solution search). So what else hidden places i could check? — ILya, May 23 '12 at 10:15
@Sander i'll give it a try. Also i will add some logging. When i get results i'll share — ILya, May 23 '12 at 10:16
Try to log values of "key" passed to FromKey(string key) method — 6opuc, May 23 '12 at 10:22
@6opuc and this logging supossed to tell? I know what is passed as a parameter. Character string like: KEVX-CD9S-K38H-4AEX-F86KR. Here is the whole source: http://pastebin.com/RU4RNCki — ILya, May 23 '12 at 10:31
If your sources are exactly as posted then this does indeed look like a bug in the runtime - either the custom setup your describe or maybe even Mono itself. — LukeH, May 23 '12 at 10:53
I've tried to proxy Dictionary as @Sander suggested. The behavior proved to be as expected. There are only TryGetValue calls. What is strange is that i get similar similar exceptions for dictionaries used inside NHibernate and even inside of System.Web classes. Please look at 1st UPD. Maybe it's related to bad linking? — ILya, May 23 '12 at 11:13
I'm not sure if this has anything to do with your problem, but you are missing some letters from your alphabet. Is this intentional? — Branko Dimitrijevic, May 23 '12 at 11:18
No, this is restricted alphabet. In any case it simply can't be related with similar issues with dictionaries in other places — ILya, May 23 '12 at 11:38
Can the user submit a key? I appreciate that leaving out i/o might be intentional to avoid confusion but what happens if a user tries ABCDE-io by accident? — ta.speot.is, May 23 '12 at 12:57

score 12 · Accepted Answer · answered May 23 '12 at 13:01

12

what i've done is statically linked mono with my executable

You already know the problem I think, that certainly would break mono. The most important thing that doesn't happen anymore when you link it statically is the DllMain() callback that Windows makes whenever a new thread starts executing in the process. That's how the CLR becomes aware of threads that may execute managed code. Static constructors are indeed a likely failure mode, a thread must be blocked from executing any code in the class until the cctor() finished executing. The CLR cannot do this if it doesn't know about the thread.

If you want to make this work then you'll at least need to provide an alternative for the mono_thread_info_attach() call in your own DllMain() function. In general, I'd say this was one optimization too many.

answered May 23 '12 at 13:01

Hans Passant

922,412
146
1,693
2,536

1

You are completely right. I've found a discussion that repeats what you say http://mono.1490590.n4.nabble.com/Mono-static-library-td3546774.html . But it leads me into another question. If i force libmono to use SGen instead of Boehm could it work then? And if yes, how can i do it... Now i'm trying to compile libmono with HAVE_SGEN_GC but a lot of external dependencies are missing... – ILya May 23 '12 at 13:57
Yes, this is a problem for GC as well. But that's not your issue, you are having a problem with static constructors. So no, SGen won't solve that problem. I don't see anybody jumping in to offer a solution in that mono page, it is certainly an unsolvable problem if you have no DLL at all. You'd better write this off as a viable option. – Hans Passant May 23 '12 at 14:51
Sure i see this is not exactly my issue. I pointed it only in terms of static linking which in fact the real cause of my issue. I think depending on its design (which i haven't investigated) SGen could solve a problem if it doesn't use DllMain which isn't very probable. In any case a windows port is still under development, so for today it's no other choise. Thanks for your useful answers and comments, Hans – ILya May 23 '12 at 18:08

Mysterious behavior of Dictionary

1 Answers1

Linked