5

I am profiling a tensorflow GPU application with NVIDIA's command line Visual Profiler nvprof, and one of the kernels that was launched and is very active in the profiling is something called redzone_checker? I cannot for the life of me find any useful information on what this means anywhere on the internet...

Any help or tips would be greatly appreciated.

redzone_checker kernel in nvprof

jakemdaly
  • 67
  • 4
  • I suppose it checks performance capacity of your device, i.e. it checks executed instructions per cycle IPC and so on and will give you an OOM error if you're running out of memory. – J.E.K Mar 24 '20 at 08:15
  • So for example, if I want the profile of my application itself, I would not include those metrics relevant to redzone_checker? ie. when my application is running without visual profiler, redzone_checker isn't there? Is there somewhere I could read more information? – jakemdaly Mar 24 '20 at 16:10

2 Answers2

0

redzone_checker kernel is implemented in TensorFlow (v2.3.0) https://github.com/tensorflow/tensorflow/blob/master/tensorflow/stream_executor/gpu/redzone_allocator.cc line 138

According to the comment in the code, the redzone_checker kernel checks that every byte in input_buffer is equal to redzone_pattern.

Sorry for uncertain information, I guess the term (redzone) is brought from the redzone on memory protection. The redzones around stack or global object to detect overflows and underflows.

I profiled using nvprof (with --print-gpu-trace option) mnist examples(https://www.tensorflow.org/xla) with XLA JIT compilation and without it. redzone_checker invocations are shown only mnist with XLA JIT compilation but there is no redzone_checker invocation in the other profiled result.

My conclusion is that modification on kernel provided by TensorFlow (even optimized by XLA compiler) leads invocations of redzone_checker to protect memory.

S.Hong
  • 1
  • 1
-1

You can try redzone/cuda?eed=0.575x

frame= 248 fps= 55 q=31.0 size= 256kB time=00:00:02.69 bitrate= 778.7kbits/s speed=0.592x
frame= 268 fps= 53 q=31.0 size= 256kB time=00:00:03.04 1
Frightera
  • 4,773
  • 2
  • 13
  • 28
Mike Coverdell
  • 1