in IIS, System.Random.NextBytes cost high cpu usage

Question

today, my IIS site take up hign cpu, and my server was very slowly! And I get a dump from task manager, then execute IISReset, question was solved.

In windbg, I use !runaway to find thread that long time executed, like: enter image description here

And use !clrstack to find out that top 10 thread were executing this code: System.Random.NextBytes(Byte[])), like:

Thread  30
Current frame: (MethodDesc 00007ffb5252c418 +0x20 System.Random.InternalSample())
Child-SP         RetAddr          Caller, Callee
000000487470b460 00007ffb531d8138 (MethodDesc 00007ffb5252c458 +0x28 System.Random.NextBytes(Byte[]))
000000487470b4b0 00007ffaffafe87a (MethodDesc 00007ffafeeb4e20 +0x4a Pinpoint.Core.Util.SpanIdUtil.GetNewSpanId())
000000487470b4e0 00007ffb014a53fb (MethodDesc 00007ffafeeb4e30 +0x1b Pinpoint.Core.Util.SpanIdUtil.GetNextSpanId(Int64, Int64))
000000487470b520 00007ffb014a5370 (MethodDesc 00007ffafeeb4150 +0x30 Pinpoint.Core.TraceId.GetNextTraceId())
000000487470b5a0 00007ffaffae4dab (MethodDesc 00007ffaff317ed0 +0x1cb Mike.Pinpoint.Plugin.WebRequestPlugins.TraceHttpWebRequest.BeforeGetResponse(System.Net.HttpWebRequest))
000000487470bbb0 00007ffb529444e0 (MethodDesc 00007ffb52535888 +0x80 System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]))
000000487470bcf0 00007ffb529444e0 (MethodDesc 00007ffb52535888 +0x80 System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]))
000000487470bd60 00007ffb5292f4ee (MethodDesc 00007ffb52535870 +0x8e System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo))
000000487470bde0 00007ffaffae4b79 (MethodDesc 00007ffaff318238 +0x99 Mike.Pinpoint.Plugin.WebRequestPlugins.TraceFilterImpl.BeforeGetResponseForFilter(System.Net.HttpWebRequest))
000000487470bdf0 00007ffb528f93e0 (MethodDesc 00007ffb526b1270 +0x120 System.IO.StreamWriter.Dispose(Boolean))

And here is my original code:

using System;

namespace Pinpoint.Core.Util
{
    public class SpanIdUtil
    {
        private static readonly Random rnd = new Random(Environment.TickCount);
        public const long Null = -1;

        public static long GetNewSpanId()
        {
            byte[] buf = new byte[8];
            rnd.NextBytes(buf);
            long newId = BitConverter.ToInt64(buf, 0);
            return newId;
        }

        public static long GetNextSpanId(long spanId, long parentSpanId)
        {
            long newId = GetNewSpanId();

            while (newId == spanId || newId == parentSpanId)
            {
                newId = GetNewSpanId();
            }
            return newId;
        }
    }
}

Can anybody tell me, what caused this error? Note, this only happened one time.

See https://stackoverflow.com/questions/3049467/is-c-sharp-random-number-generator-thread-safe — Ian Mercer, Mar 21 '19 at 05:26
thanks, but thread unsafe isn't my question, I only want a rand long num. — youbl, Mar 21 '19 at 05:40
You have a static singleton Random instance, you are calling it from multiple threads, it isn't thread safe => this is almost certainly the problem that caused your thread to hang in the while loop. — Ian Mercer, Mar 21 '19 at 05:43
en, I've seed this code: System.Random.InternalSample(), it maybe unsafe, but it will not dead lock or cost cpu! — youbl, Mar 21 '19 at 05:45
System.Random.InternalSample: this.SeedArray[index1] maybe throw exception, but my code have no exception, I think it's not my question. — youbl, Mar 21 '19 at 05:47
Who said anything about an Exception? And where did 'internalSample' or 'SeedArray' come from? Not in that link. Your code is hanging in that loop because Random has ceased being random and you loop incessantly with no check for termination. The debugger catches the threads in the Random call because that's the slowest part of your loop. — Ian Mercer, Mar 21 '19 at 05:52
I found a word at msdn， maybe this caused: If you don't ensure that the Random object is accessed in a thread-safe way, calls to methods that return random numbers return 0. — youbl, Mar 21 '19 at 05:54

score 3 · Accepted Answer · answered Mar 21 '19 at 05:32

Likely because you are sharing the same instance of Random across threads you hit a problem inside the random class as described here in which it can get stuck returning zeroes. Random is NOT thread safe.

Personally, on a server app I would NEVER use a while loop without a guard that counts down and throws an exception if it gets to zero. Unless you can prove (mathematically) that your loop will finish you should guard against it getting stuck because in a server app, when it does, you have lost a thread forever.

@youbl Yes, your threads were stuck in the loop, trying over and over to get a newId that didn't match but getting the same value over and over because Random isn't thread safe. — Ian Mercer, Mar 21 '19 at 05:46

in IIS, System.Random.NextBytes cost high cpu usage

1 Answers1