4

If many threads in a warp want to read an address in global memory, this data is broadcasted, is that right?

If many threads in a warp want to write into an address in global memory, there is a serialization, but is not possible to predict the order, is that right?

But, the first question: If many threads in a different warps, in different blocks, want to write into an address in global memory? What the GPU gonna do? Serializes all the access to this address? Is there any guarantee of data consistence?

With Hyper-Q it is possible to launch a lot of streams containing kernels. If I have a position in the memory, and a number of threads in different kernels wants to write or read this address, what the GPU gonna do? Serializes the accesses of all threads from different kernels, or does the GPU do nothing and some inconsistencies gonna happen? Is there any guarantee of data consistence when multiple kernels are reading/writing into the same address?

tomix86
  • 1,336
  • 2
  • 18
  • 29
Tiago Carneiro
  • 199
  • 1
  • 9

1 Answers1

11

It's preferred that you ask one question per question.

If many threads in a warp want to read an adress in global memory, this data is broadcasted, is that right?

Yes this is true for Fermi (CC2.0) and beyond.

If many threads in a warp want to write into an adress in global memory, there is a serialization, but is not possible to predict the order, is that right?

Correct. The order is undefined.

If many threads in a different warps, in different blocks, want to write into an adress in global memory? What the GPU gonna do? Serializes all the access to this address?

If the accesses are simultaneous, they are serialized. Again, order is undefined.

Is there any guarantee of data consistence?

Not sure what you mean by data consistence. Anyway, what else could the GPU do except serialize simultaneous writes? I'm surprised this is such a difficult concept, as there appears to me to be no obvious alternative.

If I have a possition in the memory, and a number of threads in different kernels wants to write or read this address, what the GPU gonna do? Serializes the access of all threads from different kernels, or the GPU do nothing and some inconsistences gonna happen? Is there any guarantee of data consistence when multiple kernels are reading/writing into the same address?

It does not matter what is the origin of simultaneous writes to global memory, whether from the same warp, or different warps, in different blocks, in different kernels. Simultaneous writes are serialized, in an undefined order. Again, for "data consistence" I'd like to know what you mean by that. Simultaneous reads and writes are also going to produce undefined behavior. The reads may return a value including the initial value of the memory location or any of the values that were written.

The final result of simultaneous writes to any GPU memory location is undefined. If all simultaneous writes are writing the same value, then the final value in that location will reflect that. Otherwise, the final value will reflect one of the values that got written. Which value is undefined. Beyond that, most of your questions and statements don't make sense to me. (What do you mean by data consistence?) You should not expect anything rational from such programming behavior. The GPU should be programmed as a distributed independent work machine, not a globally synchronous machine. Note that "undefined" also means that results may vary from one run of a kernel to the next, even if the input data is identical.

Simultaneous or nearly simultaneous reading and writing of global memory from different blocks (whether from the same or different kernels) is especially hazardous on Fermi (cc2.x) devices due to the independent non-coherent L1 caches that are interposed between the SMs (where the threadblocks execute) and the L2 cache (which is device-wide, and therefore coherent). Attempting to create synchronized behavior between threadblocks using global memory as a vehicle is difficult at best, and discouraged. It is suggested to consider ways to recast your algorithm to structure the work independently.

Robert Crovella
  • 143,785
  • 11
  • 213
  • 257
  • 2
    Your answer to the second point isn't really correct is it? Multiple writes to a single global memory from the same warp are only serialised if the access is done by an atomic primitive. Otherwise one (undefined) thread will win and get the write through, the others will loose. – talonmies Jan 22 '13 at 06:43
  • 1
    @talonmies the specific behavior is (intentionally) not well specified. You may be correct as to what goes on under the hood. However I think we would agree that the net effect is the same undefined behavior. Whether all writes occur or not, if the "winning" thread (if all writes do not occur) or the "final" thread (if all writes are serialized and do occur) is undefined, then from the programmer's perspective the behavior is the same. In any case of simultaneous writes, *one* of the written values will end up in the location, after the simultaneous write is processed. – Robert Crovella Jan 22 '13 at 13:23
  • "If many threads in a warp want to write into an adress in global memory, there is a serialization, but is not possible to predict the order, is that right?" _Robert's Answer: Correct. The order is undefined._ ** I have a new question: is there any guarantee that all threads will write and the final result is the value writen by the last thread?** – Tiago Carneiro Jan 23 '13 at 17:14
  • The final result is always the last value written (by the last thread, or "winning" thread). However there's no way to predict which thread will be "last". It is undefined. – Robert Crovella Jan 23 '13 at 17:29