[compiler-rt] [builtins] Align addresses to cache lines in __clear_cache for aarch64

[mstorsjo]

Summary

This makes sure that the last cache line gets invalidated properly.

This matches the example code at http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.den0024a/BABJDBHI.html.

[mstorsjo]

Ping @t.p.northover

[peter.smith]

I guess this comes down to the interpretation of (begin, end] when you can only clear a cache line at a time. I think that this makes sense as it matches the powerpc64 below, Linux also follows this approach. I'm not sure how this is done on BSD and Darwin.

Linux links:

• Implementation: http://elixir.free-electrons.com/linux/latest/source/arch/arm64/mm/cache.S
• Interface with comment "Start addresses are inclusive and end addresses are exclusive; start addresses should be rounded down, end addresses up.": https://elixir.free-electrons.com/linux/latest/source/arch/arm64/include/asm/cacheflush.h

[mstorsjo]

In D42196#984791, @peter.smith wrote:

I guess this comes down to the interpretation of (begin, end] when you can only clear a cache line at a time. I think that this makes sense as it matches the powerpc64 below, Linux also follows this approach. I'm not sure how this is done on BSD and Darwin.

It also matches libgcc: https://github.com/gcc-mirror/gcc/blob/master/libgcc/config/aarch64/sync-cache.c

I'm not sure at what granularity memory protection can be set (if it can be more granular than the cache line size) - in that case, we shouldn't align the start but instead align after each (= the first) increment instead. But seeing as nobody else does that, including libgcc that operates in user mode, I think this should be fine.

[mstorsjo]

@peter.smith Would you care to approve this one, when there don't seem to be anybody else interested in it?

[peter.smith]

Yes I'm happy to approve.

[mstorsjo]

@hans - I think this is a pretty simple/straightforward bugfix that might be worthy of backporting to 6.0.

[hans]

In D42196#986150, @mstorsjo wrote:

@hans - I think this is a pretty simple/straightforward bugfix that might be worthy of backporting to 6.0.

Merged in r323338.

compiler-rt/trunk/lib/builtins/clear_cache.c

#elif defined(__aarch64__) && !defined(__APPLE__)
   uint64_t ctr_el0;
   __asm __volatile("mrs %0, ctr_el0" : "=r"(ctr_el0));
 
   /*
    * dc & ic instructions must use 64bit registers so we don't use
    * uintptr_t in case this runs in an IPL32 environment.
    */
   const size_t dcache_line_size = 4 << ((ctr_el0 >> 16) & 15);
-  for (addr = xstart; addr < xend; addr += dcache_line_size)
+  for (addr = xstart & ~(dcache_line_size - 1); addr < xend;
+       addr += dcache_line_size)
     __asm __volatile("dc cvau, %0" :: "r"(addr));
   __asm __volatile("dsb ish");
 
   const size_t icache_line_size = 4 << ((ctr_el0 >> 0) & 15);
-  for (addr = xstart; addr < xend; addr += icache_line_size)
+  for (addr = xstart & ~(icache_line_size - 1); addr < xend;
+       addr += icache_line_size)
     __asm __volatile("ic ivau, %0" :: "r"(addr));
   __asm __volatile("isb sy");
 #elif defined (__powerpc64__)
   const size_t line_size = 32;
   const size_t len = (uintptr_t)end - (uintptr_t)start;
 
   const uintptr_t mask = ~(line_size - 1);
   const uintptr_t start_line = ((uintptr_t)start) & mask;