[AArch64][Builtins] Avoid unnecssary cache cleaning

[bryanpkc]

Summary

Use new control bits CTR_EL0.DIC and CTR_EL0.IDC to discover the d-cache
cleaning and i-cache invalidation requirements for instruction-to-data
coherence. This matches the behavior in the latest libgcc.

Author: Shaokun Zhang <[e-mail]>

[bryanpkc]

Pinging reviewers.

[peter.smith]

I'm on vacation at the moment, will take a look next week as this will need some checking of the architecture manual. This LKML message https://lkml.org/lkml/2018/10/9/698 implies that in some cases it is insufficient to just read CTR_EL0.{IDC,DIC}, as a 0 also requires a check with CTR_EL1 which can't be done in userspace. Would you be able to take a look at that thread? For compiler-rt we need something that works on all CPUs at EL0.

[ktkachov]

In D69247#1719597, @peter.smith wrote:

I'm on vacation at the moment, will take a look next week as this will need some checking of the architecture manual. This LKML message https://lkml.org/lkml/2018/10/9/698 implies that in some cases it is insufficient to just read CTR_EL0.{IDC,DIC}, as a 0 also requires a check with CTR_EL1 which can't be done in userspace. Would you be able to take a look at that thread? For compiler-rt we need something that works on all CPUs at EL0.

The way I read that LKML discussion is that it's a kernel discussion on how to expose this feature to userspace (EL0). On earlier architecture revisions this field is architected as "reserved zero", meaning reads from it will be 0. Therefore at boot time the kernel needs to look at a higher EL to figure out if the feature actually exists. Once it figures that out it will set the EL0 field (probably through trapping) to the appropriate value for userspace, which is what we should be reading. Note that a value of 0 here is conservatively correct: it means userspace has to do the explicit cache clean operation, which is always safe to do in this context. This change tries to figure when it's safe to avoid that cache clean.

This logic is the same as the libgcc logic in http://gcc.gnu.org/r276122 so it looks good to me. The libgcc implementation caches the value of ctr_el0 in a static variable to avoid a sysreg read on every call. Can this be done here as well?

[bryanpkc]

I actually read that LKML post before submitting the patch, and I had the same understanding as @ktkachov. I will change the code to cache the result of the MRS instruction as suggested.

[bryanpkc]

Avoid reading CTR_EL0 on every call to __clear_cache.

[peter.smith]

Thanks both for the extra information and update. Looks good to me.

lib/builtins/clear_cache.c

#elif defined(__linux__) && defined(__mips__)
   const uintptr_t start_int = (uintptr_t)start;
   const uintptr_t end_int = (uintptr_t)end;
   syscall(__NR_cacheflush, start, (end_int - start_int), BCACHE);
 #elif defined(__mips__) && defined(__OpenBSD__)
   cacheflush(start, (uintptr_t)end - (uintptr_t)start, BCACHE);
 #elif defined(__aarch64__) && !defined(__APPLE__)
   uint64_t xstart = (uint64_t)(uintptr_t)start;
   uint64_t xend = (uint64_t)(uintptr_t)end;
-  uint64_t addr;
 
-  // Get Cache Type Info
+  // Get Cache Type Info.
   uint64_t ctr_el0;
   __asm __volatile("mrs %0, ctr_el0" : "=r"(ctr_el0));
 
-  // dc & ic instructions must use 64bit registers so we don't use
+  // The DC and IC instructions must use 64-bit registers so we don't use
   // uintptr_t in case this runs in an IPL32 environment.
+  uint64_t addr;
+
+  // If CTR_EL0.IDC is set, data cache cleaning to the point of unification
+  // is not required for instruction to data coherence.
+  if (((ctr_el0 >> 28) & 0x1) == 0x0) {
     const size_t dcache_line_size = 4 << ((ctr_el0 >> 16) & 15);
     for (addr = xstart & ~(dcache_line_size - 1); addr < xend;
          addr += dcache_line_size)
       __asm __volatile("dc cvau, %0" ::"r"(addr));
+  }
   __asm __volatile("dsb ish");
 
+  // If CTR_EL0.DIC is set, instruction cache invalidation to the point of
+  // unification is not required for instruction to data coherence.
+  if (((ctr_el0 >> 29) & 0x1) == 0x0) {
     const size_t icache_line_size = 4 << ((ctr_el0 >> 0) & 15);
     for (addr = xstart & ~(icache_line_size - 1); addr < xend;
          addr += icache_line_size)
       __asm __volatile("ic ivau, %0" ::"r"(addr));
+  }
   __asm __volatile("isb sy");
 #elif defined(__powerpc64__)
   const size_t line_size = 32;
   const size_t len = (uintptr_t)end - (uintptr_t)start;
 
   const uintptr_t mask = ~(line_size - 1);
   const uintptr_t start_line = ((uintptr_t)start) & mask;
   const uintptr_t end_line = ((uintptr_t)start + len + line_size - 1) & mask;