Can coherency issue happen between secure DMA and non-secure CPU on TrustZone system

Question

I encounter some problem which I think is about coherency between DMA and CPU. Here is the simplified use case.

1. Cortex A5 CPU writes to the non-secure memory under non-secure state. MMU is enabled and the memory attribute is normal, shareable and cacheable. So the data may be in so-called non-secure cache according to the following references:

   [1] "Data Cache Tag data format" in DDI0434B_cortex_a5_mpcore_r0p1_trm

   [2] Answer from artless noise in Secure mode can access secure / non secure memory how?

   [3] 6th page in http://atmel.force.com/support/servlet/fileField?id=0BEG000000002Ur

2. DMA issues a secure coherent read to that memory area. DMA is connected to ACP.

3. SCU will only try to check cache tagged as secure. So the data is returned from main memory according to "ACP requests" in DDI0434B_cortex_a5_mpcore_r0p1_trm.

4. DMA may get stale data.

---

If my description is correct, here is my second question. Which action can solve this issue?

1) Clean the cache explicitly before DMA reads

2) Change the attribute of the memory to normal, shareable and non-cacheable.

Also the TRM says there is a write buffer that is used to hold data from cache evictions or non-cacheable write bursts before they are written out on the SCU interface in "Bus Interface Unit and SCU interface". However no more information about the write buffer can be found. May I assume the write buffer has some magic to make sure coherency? I would appreciate a lot if someone can explain the magic.

---

Considering more combinations, I get this table myself.

 CPU state |   Memory   | DMA access | Result    
-----------+------------+------------+-----------+    
non-secure | non-secure | non-secure | OK            
non-secure | non-secure |   secure   | NG    
non-secure |   secure   |     -      | NA    
secure     | non-secure | non-secure | OK    
secure     | non-secure |   secure   | NG (Same reason as the second case)
secure     |   secure   | non-secure | NA    
secure     |   secure   |   secure   | OK

The first column shows under which state CPU writes to the memory.

The second column shows the memory is secure or not. Here "secure" means access will be denied by AXI if the AxPROT[1] is high. Memory attribute is normal, shareable and cacheable.

The third column shows what kinds of access DMA issues. The access is coherent.

The fourth column tells what will happen.

NG means DMA may get stale data.

NA means the access is impossible.

OK means DMA is coherent with CPU.

Is is right?

---

I add this part according to Notlikethat's answer. It may be a nightmare in practice. However I want to know what happens theoretically.

1. Suppose that the physical address 0x10 is mapped to two virtual addresses, NS:0x110 and S:0x210.
2. CPU under secure state writes value 0x110 to NS:0x110, and then writes value 0x210 to S:0x210.
   So both may be in L1 cache at the same time, right?
3. Then DMA1 makes a secure read to PA 0x10 and may get 0x210. DMA2 makes a non-secure read to PA 0x10 and may get 0x110.
4. After that it is unpredictable which one will be in the main memory eventually.

Please confirm or correct my understandings. Thank a lot.

Answer 1

Architecturally, you can't make "secure accesses to non-secure memory". The secure and non-secure physical address spaces are independent (many TrustZone-capable systems have things which appear at different addresses to secure vs. non-secure accesses). In practice though, it is the case that a lot of peripherals (including memory controllers) aren't TrustZone-aware, thus on most systems are behind a TZASC at which point the secure and non-secure address spaces do largely overlap.

On the CPU, secure software accesses non-secure memory either via a page table entry with the NS bit set, or from monitor mode with SCR.NS set; in both cases, the resulting bus access has AxPROT[1] set, i.e. it is a non-secure access. If you want a peripheral to correctly access non-secure memory, you need to get it to issue non-secure accesses in a similar fashion.

Otherwise, what you have is a case of the more general problem of physical aliases, which are usually not recommended unless absolutely necessary - the more traditional example is systems with DRAM at fairly high physical addresses (e.g. 0x80000000, or even above 32 bits on LPAE-capable systems), part of which is aliased lower down (e.g. at 0x0 for boot vectors) for use until the MMU is set up. Even a physically-tagged cache has no idea that 0x0 and 0x80000000 actually end up at the same place on the other end of the interconnect, so if you were to use both at once you would indeed have a coherency nightmare which is outside the scope of the architecture, and can only be managed with explicit cache maintenance.

In the same manner, a TrustZone-aware cache, which includes the security state as part of the physical address tag, has no idea that S:0x80000000 and NS:0x80000000 might actually end up in the same place on your particular system (in general, they might well not), so again, the two addresses are not coherent unless managed manually - data written to one alias must be cleaned from the cache to a point beyond the TZASC (i.e. usually all the way to DRAM) before it is visible from the other. Note that if your Cortex-A5 system has an outer L2 cache like a PL310, that means cleaning the CPU caches by VA to PoC, followed by cleaning L2 by PA via secure/non-secure accesses as appropriate, which probably all has to be done by the secure world alone to avoid synchronisation problems. In theory having everything make non-cacheable accesses would work around the coherency issue by forcing all data to take the round trip through DRAM, although it's not impossible that certain outer cache configurations could still get in the way of that. Far better to make the DMA controller issue non-secure accesses directly where appropriate, so you can actually benefit from the caches rather than fight against them.