[ASan] Improved stack overflow detection for PowerPC64

[foad]

Summary

AsanOnSIGSEGV has some heuristics for detecting stack overflow, but
they don't cope with a PowerPC store-with-update instruction which
modifies sp and stores to the modified address in one instruction.

This patch adds some PowerPC-specific code to check for this case.

This fixes the last few cases of the stack-overflow test.

[eugenis]

stack-overflow test actually catches this? Nice.

lib/asan/asan_posix.cc

57

Please make it u32 instead of unsigned.

61

It may be helpful to list or somehow characterize instructions that match this condition (I mean, there is obviously more than 1 opcode listed here).

I assume these "xo" checks limit it to instructions operating on SP register?

[foad]

lib/asan/asan_posix.cc

61

I characterized them as "store-with-update to r1" instructions, but I will be more explicit!

[foad]

lib/asan/asan_posix.cc

61

opcd is "opcode" and xo is "extended opcode". These field names are taken directly from the POWER architecture manual.

[foad]

Use u32 instead of unsigned. Add comments about which instructions
we match.

[eugenis]

LGTM

lib/asan/asan_posix.cc

void AsanOnSIGSEGV(int, void *siginfo, void *context) {
   // Write the first message using the bullet-proof write.
   if (13 != internal_write(2, "ASAN:SIGSEGV\n", 13)) Die();
   uptr pc, sp, bp;
   GetPcSpBp(context, &pc, &sp, &bp);
 
   // Access at a reasonable offset above SP, or slightly below it (to account
   // for x86_64 or PowerPC redzone, ARM push of multiple registers, etc) is
   // probably a stack overflow.
+  bool IsStackAccess = addr + 512 > sp && addr < sp + 0xFFFF;
+
+#if __powerpc64__
+  // Large stack frames can be allocated with e.g.
+  //   lis r0,-10000
+  //   stdux r1,r1,r0 # store sp to [sp-10000] and update sp by -10000
+  // If the store faults then sp will not have been updated, so test above
+  // will not work, becase the fault address will be more than just "slightly"
+  // below sp.
+  if (!IsStackAccess && IsAccessibleMemoryRange(pc, 4)) {
+    u32 inst = *(unsigned *)pc;
+    u32 ra = (inst >> 16) & 0x1F;
+    u32 opcd = inst >> 26;
+    u32 xo = (inst >> 1) & 0x3FF;
+    // Check for store-with-update to sp. The instructions we accept are:
+    //   stbu rs,d(ra)          stbux rs,ra,rb
+    //   sthu rs,d(ra)          sthux rs,ra,rb
+    //   stwu rs,d(ra)          stwux rs,ra,rb
+    //   stdu rs,ds(ra)         stdux rs,ra,rb
+    // where ra is r1 (the stack pointer).
+    if (ra == 1 &&
+        (opcd == 39 || opcd == 45 || opcd == 37 || opcd == 62 ||
+         (opcd == 31 && (xo == 247 || xo == 439 || xo == 183 || xo == 181))))
+      IsStackAccess = true;
+  }
+#endif // __powerpc64__
+
   // We also check si_code to filter out SEGV caused by something else other
   // then hitting the guard page or unmapped memory, like, for example,
   // unaligned memory access.
-  if (addr + 512 > sp && addr < sp + 0xFFFF &&
-      (code == si_SEGV_MAPERR || code == si_SEGV_ACCERR))
+  if (IsStackAccess && (code == si_SEGV_MAPERR || code == si_SEGV_ACCERR))
     ReportStackOverflow(pc, sp, bp, context, addr);
   else
     ReportSIGSEGV("SEGV", pc, sp, bp, context, addr);
 }
 
 // ---------------------- TSD ---------------- {{{1
 
 static pthread_key_t tsd_key;