Align i128 to 16 bytes

[nagisa]

Summary

This is what clang does as can be seen in [1]. As Clang does not support using 128-bit integers on
non-64 bit targets, no effort was spent on updating the data-layouts for 32-bit only backends.

This is wanted for rustc, which wants data-layout used by LLVM and data-layout used by rustc
match to some extent.

[1]: https://github.com/llvm-mirror/clang/blob/916645c2637f8c80948eedcdea02258d0a6f79f5/lib/AST/ASTContext.cpp#L1734-L1738

[chfast]

Does this affects also bigger types like i256?

[nagisa]

Does this affects also bigger types like i256?

Yes, it does, as LLVM will pick the alignment information of the largest specified integer. So now that largest specified integer type is i128, and not i64 as it was previously, i256, i512, i1024, etc types will be using alignment of i128. This is also evidenced by the test changes to mul-i{512,1024} tests for x86 backend.

[joerg]

Why? At least for some x86 targets, this will unncessarily penalize the stack. Don't know all the other 32bit architectures, but many of those don't have a stack alignment larger than 32bit either.

[nagisa]

At least for some x86 targets, this will unncessarily penalize the stack. Don't know all the other 32bit architectures, but many of those don't have a stack alignment larger than 32bit either.

I’m willing to adjust the PR to only touch 64-bit targets. For the 64-bit targets 128-bit alignment is necessary for correct FFI with Clang-generated code, which generates code with assumption that i128 is 128-aligned always.

[nagisa]

Make alignment changes specific to 64-bit targets.

[chfast]

Will not that penalize stack usage on 64-bit targets as well?

[joerg]

What is the motivation behind requiring 128bit alignment? E.g. normal lowering of __uint128_t arithmetic on x86_64 will use uint64_t arithmetic and naturally require the same alignment.

I'm concerned about 32bit platforms because I really would like to see the artifical restriction of __uint128_t to 64bit targets be removed at some point, so that code can naturally depend on them everywhere.

[majnemer]

In D28990#652740, @joerg wrote:

What is the motivation behind requiring 128bit alignment? E.g. normal lowering of __uint128_t arithmetic on x86_64 will use uint64_t arithmetic and naturally require the same alignment.

I'm concerned about 32bit platforms because I really would like to see the artifical restriction of __uint128_t to 64bit targets be removed at some point, so that code can naturally depend on them everywhere.

The x86_64 psABI specifies an alignment of 16 bytes for int128_t and uint128_t.

[nagisa]

Its as @majnemer says, 16-byte alignment is required by ABI. Not just targets which use SysV, but any, where int128 is supported, because clang hardcodes int128 to have alignment of 16-bytes on *all* targets.

I’m surprised y'all are debating efficiency when its a plain correctness issue. IMHO what should be done is having the correctness issue fixed (this patch), then maybe some work towards removing the special case from clang (in the code I linked above) and finally incremental efficiency improvements where they are applicable.

[joerg]

You make it sound as if the current alignment decision was made intentional. I strongly doubt that. So yes, please double check all the ABIs here to make sure that the wild card code is actually correct.

[nagisa]

please double check all the ABIs here

Sorry, I do not have the time on my hands to go and find and parse ABI documents for literally every target triplet LLVM supports, so here’s what I know instead:

• x86_64 SysV - 16 byte aligned;
• x64 (msvc) ABI - 16 byte aligned;
• ARM - 16 byte aligned;
• PPC64 - quadword, so 16 byte;

Sounds intentional enough to me at least.

[majnemer]

Yes, I think this LGTM if you can find rationale in the other psABI documents. https://www.uclibc.org/docs/psABI-ppc64.pdf gives the PPC64 int128 alignment as 16 bytes. Can you find justifications for i128 for the other platforms?

[nagisa]

This is for ARM: http://infocenter.arm.com/help/topic/com.arm.doc.ihi0055c/IHI0055C_beta_aapcs64.pdf;
This is for x64: https://msdn.microsoft.com/en-us/library/zthk2dkh.aspx (protip: needs careful reading because __int128 is not supported by MSVC directly and is not explicitly mentioned by the docs, but wording is unambiguous enough to make conclusions);
SPARC (https://docs.oracle.com/cd/E26502_01/pdf/E28387.pdf) does not mention anything really and there’s some encryption involving “slots” and whatnot in that document, but

For 64-bit code, double and long long datat types occupy just one slot, but long double and double complex data occupy two slots, and these slots are aligned (slot number modulo 2 equals 0), skipping a slot if necessary for alignment.

seems to indicate that stuff ought to be aligned to 2 times of whatever slots ought to mean;

[echristo]

This LGTM as well. At the very least I believe it is intentional. This is, at a minimum, correct for x86-64 and ppc64.

[nagisa]

When can I expect this to land into the trunk?

[echristo]

When you mention that you need someone to land it for you. I'll do it shortly.

[mehdi_amini]

In D28990#659198, @nagisa wrote:

When can I expect this to land into the trunk?

You need to ask for it, reviewers usually assume you have commit right unless mentioned otherwise. So when someone approves your patch just ask "can you land this for me please".

[echristo]

This patch didn't pass either the llvm testsuite or the clang testsuite. I've worked up patches for both, but there are a couple of places I think we're changing the ABI (via changes in clang that I had to make) that I'm a bit more uncomfortable with - I'll look at this over the next couple of days.

[echristo]

OK. I've committed x86-64 linux and ppc64 linux support of this patch. We should get additional confirmation on the other platforms before we change things. I believe it's correct in all cases, but I'm going to be a bit conservative here.

Thanks!

-eric

[vchuravy]

@echristo I noticed that you reverted r294702 shortly after it landed. What needs to be done to get it working?

[echristo]

In D28990#801641, @vchuravy wrote:

@echristo I noticed that you reverted r294702 shortly after it landed. What needs to be done to get it working?

From the revert:

commit 8c54b816e554ff9f9a75ec8a1d50198d2bfd9633
Author: Eric Christopher <[e-mail]>
Date: Fri Feb 10 04:35:32 2017 +0000

Temporarily revert "For X86-64 linux and PPC64 linux align int128 to 16 bytes."
until we can get better TargetMachine::isCompatibleDataLayout to compare - otherwise
we can't code generate existing bitcode without a string equality data layout.

This reverts commit r294702.

git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@294709 91177308-0d34-0410-b5e6-96231b3b80d8

lib/Target/Mips/MipsTargetMachine.cpp

else
     Ret += "-m:e";
 
   // Pointers are 32 bit on some ABIs.
   if (!ABI.IsN64())
     Ret += "-p:32:32";
 
   // 8 and 16 bit integers only need to have natural alignment, but try to
   // align them to 32 bits. 64 bit integers have natural alignment.
-  Ret += "-i8:8:32-i16:16:32-i64:64";
+  // 128 bit integers are always aligned to 128 bits.
+  Ret += "-i8:8:32-i16:16:32-i64:64-i128:128";
 
   // 32 bit registers are always available and the stack is at least 64 bit
   // aligned. On N64 64 bit registers are also available and the stack is
   // 128 bit aligned.
   if (ABI.IsN64() || ABI.IsN32())
     Ret += "-n32:64-S128";
   else
     Ret += "-n32-S64";

lib/Target/PowerPC/PPCTargetMachine.cpp

static std::string getDataLayoutString(const Triple &T) {
 
   // Note, the alignment values for f64 and i64 on ppc64 in Darwin
   // documentation are wrong; these are correct (i.e. "what gcc does").
   if (is64Bit || !T.isOSDarwin())
     Ret += "-i64:64";
   else
     Ret += "-f64:32:64";
 
+  // i128 is considered to always be aligned to 16 bytes
+  Ret += "-i128:128";
+
   // PPC64 has 32 and 64 bit registers, PPC32 has only 32 bit ones.
   if (is64Bit)
     Ret += "-n32:64";
   else
     Ret += "-n32";
 
   return Ret;
 }

lib/Target/Sparc/SparcTargetMachine.cpp

static std::string computeDataLayout(const Triple &T, bool is64Bit) {
   // Sparc is typically big endian, but some are little.
   std::string Ret = T.getArch() == Triple::sparcel ? "e" : "E";
   Ret += "-m:e";
 
   // Some ABIs have 32bit pointers.
   if (!is64Bit)
     Ret += "-p:32:32";
 
-  // Alignments for 64 bit integers.
-  Ret += "-i64:64";
+  // Alignments for 64 bit and 128 bit integers.
+  Ret += "-i64:64-i128:128";
 
   // On SparcV9 128 floats are aligned to 128 bits, on others only to 64.
   // On SparcV9 registers can hold 64 or 32 bits, on others only 32.
   if (is64Bit)
     Ret += "-n32:64";
   else
     Ret += "-f128:64-n32";
 

lib/Target/SystemZ/SystemZTargetMachine.cpp

static std::string computeDataLayout(const Triple &TT, StringRef CPU,
   // Make sure that global data has at least 16 bits of alignment by
   // default, so that we can refer to it using LARL.  We don't have any
   // special requirements for stack variables though.
   Ret += "-i1:8:16-i8:8:16";
 
   // 64-bit integers are naturally aligned.
   Ret += "-i64:64";
 
+  // 128-bit integers are always aligned to 16 bytes.
+  Ret += "-i128:128";
+
   // 128-bit floats are aligned only to 64 bits.
   Ret += "-f128:64";
 
   // When using the vector ABI, 128-bit vectors are also aligned to 64 bits.
   if (VectorABI)
     Ret += "-v128:64";
 
   // We prefer 16 bits of aligned for all globals; see above.

lib/Target/X86/X86TargetMachine.cpp

static std::string computeDataLayout(const Triple &TT) {
   // Some ABIs align 64 bit integers and doubles to 64 bits, others to 32.
   if (TT.isArch64Bit() || TT.isOSWindows() || TT.isOSNaCl())
     Ret += "-i64:64";
   else if (TT.isOSIAMCU())
     Ret += "-i64:32-f64:32";
   else
     Ret += "-f64:32:64";
 
+  // i128 is considered to always be 16-byte aligned
+  Ret += "-i128:128";
+
   // Some ABIs align long double to 128 bits, others to 32.
   if (TT.isOSNaCl() || TT.isOSIAMCU())
     ; // No f80
   else if (TT.isArch64Bit() || TT.isOSDarwin())
     Ret += "-f80:128";
   else
     Ret += "-f80:32";
 

test/CodeGen/Mips/llvm-ir/add.ll

; ALL-LABEL: add_i64:
   %r = add i64 %a, %b
   ret i64 %r
 }
 
 define signext i128 @add_i128(i128 signext %a, i128 signext %b) {
 entry:
 ; ALL-LABEL: add_i128:
 
-  ; GP32:       lw        $[[T0:[0-9]+]], 28($sp)
+  ; GP32:       lw        $[[T0:[0-9]+]], 44($fp)
   ; GP32:       addu      $[[T1:[0-9]+]], $7, $[[T0]]
   ; GP32:       sltu      $[[T2:[0-9]+]], $[[T1]], $[[T0]]
-  ; GP32:       lw        $[[T3:[0-9]+]], 24($sp)
+  ; GP32:       lw        $[[T3:[0-9]+]], 40($fp)
   ; GP32:       addu      $[[T4:[0-9]+]], $[[T2]], $[[T3]]
   ; GP32:       addu      $[[T5:[0-9]+]], $6, $[[T4]]
   ; GP32:       sltu      $[[T6:[0-9]+]], $[[T5]], $[[T3]]
-  ; GP32:       lw        $[[T7:[0-9]+]], 20($sp)
+  ; GP32:       lw        $[[T7:[0-9]+]], 36($fp)
   ; GP32:       addu      $[[T8:[0-9]+]], $[[T6]], $[[T7]]
-  ; GP32:       lw        $[[T9:[0-9]+]], 16($sp)
+  ; GP32:       lw        $[[T9:[0-9]+]], 32($fp)
   ; GP32:       addu      $3, $5, $[[T8]]
   ; GP32:       sltu      $[[T10:[0-9]+]], $3, $[[T7]]
   ; GP32:       addu      $[[T11:[0-9]+]], $[[T10]], $[[T9]]
   ; GP32:       addu      $2, $4, $[[T11]]
   ; GP32:       move      $4, $[[T5]]
   ; GP32:       move      $5, $[[T1]]
 
   ; GP64:       daddu     $3, $5, $7
   ; GP64:       sltu      $[[T0:[0-9]+]], $3, $7
   ; GP64:       daddu     $[[T1:[0-9]+]], $[[T0]], $6
   ; GP64:       daddu     $2, $4, $[[T1]]
 
-  ; MM32:       lw        $[[T0:[0-9]+]], 28($sp)
+  ; MM32:       lw        $[[T0:[0-9]+]], 44($fp)
   ; MM32:       addu      $[[T1:[0-9]+]], $7, $[[T0]]
   ; MM32:       sltu      $[[T2:[0-9]+]], $[[T1]], $[[T0]]
-  ; MM32:       lw        $[[T3:[0-9]+]], 24($sp)
+  ; MM32:       lw        $[[T3:[0-9]+]], 40($fp)
   ; MM32:       addu      $[[T4:[0-9]+]], $[[T2]], $[[T3]]
   ; MM32:       addu      $[[T5:[0-9]+]], $6, $[[T4]]
   ; MM32:       sltu      $[[T6:[0-9]+]], $[[T5]], $[[T3]]
-  ; MM32:       lw        $[[T7:[0-9]+]], 20($sp)
+  ; MM32:       lw        $[[T7:[0-9]+]], 36($fp)
   ; MM32:       addu      $[[T8:[0-9]+]], $[[T6]], $[[T7]]
-  ; MM32:       lw        $[[T9:[0-9]+]], 16($sp)
+  ; MM32:       lw        $[[T9:[0-9]+]], 32($fp)
   ; MM32:       addu      $[[T10:[0-9]+]], $5, $[[T8]]
   ; MM32:       sltu      $[[T11:[0-9]+]], $[[T10]], $[[T7]]
   ; MM32:       addu      $[[T12:[0-9]+]], $[[T11]], $[[T9]]
   ; MM32:       addu      $[[T13:[0-9]+]], $4, $[[T12]]
   ; MM32:       move      $4, $[[T5]]
   ; MM32:       move      $5, $[[T1]]
 
   ; MM64:       daddu     $3, $5, $7

test/CodeGen/Mips/llvm-ir/and.ll

; ALL-LABEL: and_i64:
   %r = and i64 %a, %b
   ret i64 %r
 }
 
 define signext i128 @and_i128(i128 signext %a, i128 signext %b) {
 entry:
 ; ALL-LABEL: and_i128:
 
-  ; GP32:         lw      $[[T0:[0-9]+]], 20($sp)
-  ; GP32:         lw      $[[T1:[0-9]+]], 16($sp)
+  ; GP32:         lw      $[[T0:[0-9]+]], 36($fp)
+  ; GP32:         lw      $[[T1:[0-9]+]], 32($fp)
   ; GP32:         and     $2, $4, $[[T1]]
   ; GP32:         and     $3, $5, $[[T0]]
-  ; GP32:         lw      $[[T2:[0-9]+]], 24($sp)
+  ; GP32:         lw      $[[T2:[0-9]+]], 40($fp)
   ; GP32:         and     $4, $6, $[[T2]]
-  ; GP32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; GP32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; GP32:         and     $5, $7, $[[T3]]
 
   ; GP64:         and     $2, $4, $6
   ; GP64:         and     $3, $5, $7
 
-  ; MM32:         lw      $[[T0:[0-9]+]], 20($sp)
-  ; MM32:         lw      $[[T1:[0-9]+]], 16($sp)
+  ; MM32:         lw      $[[T0:[0-9]+]], 36($fp)
+  ; MM32:         lw      $[[T1:[0-9]+]], 32($fp)
   ; MM32:         and16   $[[T1]], $4
   ; MM32:         and16   $[[T0]], $5
-  ; MM32:         lw      $[[T2:[0-9]+]], 24($sp)
+  ; MM32:         lw      $[[T2:[0-9]+]], 40($fp)
   ; MM32:         and16   $[[T2]], $6
-  ; MM32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; MM32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; MM32:         and16   $[[T3]], $7
 
   ; MM64:         and     $2, $4, $6
   ; MM64:         and     $3, $5, $7
 
   %r = and i128 %a, %b
   ret i128 %r
 }

test/CodeGen/Mips/llvm-ir/not.ll

; ALL-LABEL: nor_i64:
   %r = xor i64 %or, -1
   ret i64 %r
 }
 
 define signext i128 @nor_i128(i128 signext %a, i128 signext %b) {
 entry:
 ; ALL-LABEL: nor_i128:
 
-  ; GP32:         lw      $[[T1:[0-9]+]], 20($sp)
-  ; GP32:         lw      $[[T2:[0-9]+]], 16($sp)
+  ; GP32:         lw      $[[T1:[0-9]+]], 36($fp)
+  ; GP32:         lw      $[[T2:[0-9]+]], 32($fp)
   ; GP32:         nor     $2, $[[T2]], $4
   ; GP32:         nor     $3, $[[T1]], $5
-  ; GP32:         lw      $[[T0:[0-9]+]], 24($sp)
+  ; GP32:         lw      $[[T0:[0-9]+]], 40($fp)
   ; GP32:         nor     $4, $[[T0]], $6
-  ; GP32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; GP32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; GP32:         nor     $5, $[[T3]], $7
 
   ; GP64:         nor     $2, $6, $4
   ; GP64:         nor     $3, $7, $5
 
-  ; MM32:         lw      $[[T1:[0-9]+]], 20($sp)
-  ; MM32:         lw      $[[T2:[0-9]+]], 16($sp)
+  ; MM32:         lw      $[[T1:[0-9]+]], 36($fp)
+  ; MM32:         lw      $[[T2:[0-9]+]], 32($fp)
   ; MM32:         nor     $2, $[[T2]], $4
   ; MM32:         nor     $3, $[[T1]], $5
-  ; MM32:         lw      $[[T0:[0-9]+]], 24($sp)
+  ; MM32:         lw      $[[T0:[0-9]+]], 40($fp)
   ; MM32:         nor     $4, $[[T0]], $6
-  ; MM32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; MM32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; MM32:         nor     $5, $[[T3]], $7
 
   ; MM64:         nor     $2, $6, $4
   ; MM64:         nor     $3, $7, $5
 
   %or = or i128 %b, %a
   %r = xor i128 %or, -1
   ret i128 %r
 }

test/CodeGen/Mips/llvm-ir/or.ll

; ALL-LABEL: or_i64:
   %r = or i64 %a, %b
   ret i64 %r
 }
 
 define signext i128 @or_i128(i128 signext %a, i128 signext %b) {
 entry:
 ; ALL-LABEL: or_i128:
 
-  ; GP32:         lw      $[[T1:[0-9]+]], 20($sp)
-  ; GP32:         lw      $[[T2:[0-9]+]], 16($sp)
+  ; GP32:         lw      $[[T1:[0-9]+]], 36($fp)
+  ; GP32:         lw      $[[T2:[0-9]+]], 32($fp)
   ; GP32:         or      $2, $4, $[[T2]]
   ; GP32:         or      $3, $5, $[[T1]]
-  ; GP32:         lw      $[[T0:[0-9]+]], 24($sp)
+  ; GP32:         lw      $[[T0:[0-9]+]], 40($fp)
   ; GP32:         or      $4, $6, $[[T0]]
-  ; GP32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; GP32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; GP32:         or      $5, $7, $[[T3]]
 
   ; GP64:         or      $2, $4, $6
   ; GP64:         or      $3, $5, $7
 
-  ; MM32:         lw      $[[T1:[0-9]+]], 20($sp)
-  ; MM32:         lw      $[[T2:[0-9]+]], 16($sp)
+  ; MM32:         lw      $[[T1:[0-9]+]], 36($fp)
+  ; MM32:         lw      $[[T2:[0-9]+]], 32($fp)
   ; MM32:         or16    $[[T2]], $4
   ; MM32:         or16    $[[T1]], $5
-  ; MM32:         lw      $[[T0:[0-9]+]], 24($sp)
+  ; MM32:         lw      $[[T0:[0-9]+]], 40($fp)
   ; MM32:         or16    $[[T0]], $6
-  ; MM32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; MM32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; MM32:         or16    $[[T3]], $7
 
   ; MM64:         or      $2, $4, $6
   ; MM64:         or      $3, $5, $7
 
   %r = or i128 %a, %b
   ret i128 %r
 }

test/CodeGen/Mips/llvm-ir/sub.ll

; ALL-LABEL: sub_i64:
   %r = sub i64 %a, %b
   ret i64 %r
 }
 
 define signext i128 @sub_i128(i128 signext %a, i128 signext %b) {
 entry:
 ; ALL-LABEL: sub_i128:
 
-  ; GP32-NOT-MM:    lw        $[[T0:[0-9]+]], 20($sp)
+  ; GP32-NOT-MM:    lw        $[[T0:[0-9]+]], 36($fp)
   ; GP32-NOT-MM:    sltu      $[[T1:[0-9]+]], $5, $[[T0]]
-  ; GP32-NOT-MM:    lw        $[[T2:[0-9]+]], 16($sp)
+  ; GP32-NOT-MM:    lw        $[[T2:[0-9]+]], 32($fp)
   ; GP32-NOT-MM:    addu      $[[T3:[0-9]+]], $[[T1]], $[[T2]]
-  ; GP32-NOT-MM:    lw        $[[T4:[0-9]+]], 24($sp)
-  ; GP32-NOT-MM:    lw        $[[T5:[0-9]+]], 28($sp)
+  ; GP32-NOT-MM:    lw        $[[T4:[0-9]+]], 40($fp)
+  ; GP32-NOT-MM:    lw        $[[T5:[0-9]+]], 44($fp)
   ; GP32-NOT-MM:    subu      $[[T6:[0-9]+]], $7, $[[T5]]
   ; GP32-NOT-MM:    subu      $2, $4, $[[T3]]
   ; GP32-NOT-MM:    sltu      $[[T8:[0-9]+]], $6, $[[T4]]
   ; GP32-NOT-MM:    addu      $[[T9:[0-9]+]], $[[T8]], $[[T0]]
   ; GP32-NOT-MM:    subu      $3, $5, $[[T9]]
   ; GP32-NOT-MM:    sltu      $[[T10:[0-9]+]], $7, $[[T5]]
   ; GP32-NOT-MM:    addu      $[[T11:[0-9]+]], $[[T10]], $[[T4]]
   ; GP32-NOT-MM:    subu      $4, $6, $[[T11]]
   ; GP32-NOT-MM:    move      $5, $[[T6]]
 
-  ; GP32-MM:        lw        $[[T0:[0-9]+]], 20($sp)
+  ; GP32-MM:        lw        $[[T0:[0-9]+]], 36($fp)
   ; GP32-MM:        sltu      $[[T1:[0-9]+]], $[[T2:[0-9]+]], $[[T0]]
-  ; GP32-MM:        lw        $[[T3:[0-9]+]], 16($sp)
+  ; GP32-MM:        lw        $[[T3:[0-9]+]], 32($fp)
   ; GP32-MM:        addu      $[[T3]], $[[T1]], $[[T3]]
-  ; GP32-MM:        lw        $[[T4:[0-9]+]], 24($sp)
-  ; GP32-MM:        lw        $[[T5:[0-9]+]], 28($sp)
+  ; GP32-MM:        lw        $[[T4:[0-9]+]], 40($fp)
+  ; GP32-MM:        lw        $[[T5:[0-9]+]], 44($fp)
   ; GP32-MM:        subu      $[[T1]], $7, $[[T5]]
   ; GP32-MM:        subu      $[[T3]], $[[T6:[0-9]+]], $[[T3]]
   ; GP32-MM:        sltu      $[[T6]], $6, $[[T4]]
   ; GP32-MM:        addu      $[[T0]], $[[T6]], $[[T0]]
   ; GP32-MM:        subu      $[[T0]], $5, $[[T0]]
   ; GP32-MM:        sltu      $[[T6]], $7, $[[T5]]
   ; GP32-MM:        addu      $[[T6]], $[[T6]], $[[T4]]
   ; GP32-MM:        subu      $[[T6]], $6, $[[T6]]

test/CodeGen/Mips/llvm-ir/xor.ll

; ALL-LABEL: xor_i64:
   %r = xor i64 %a, %b
   ret i64 %r
 }
 
 define signext i128 @xor_i128(i128 signext %a, i128 signext %b) {
 entry:
 ; ALL-LABEL: xor_i128:
 
-  ; GP32:         lw      $[[T1:[0-9]+]], 20($sp)
-  ; GP32:         lw      $[[T2:[0-9]+]], 16($sp)
+  ; GP32:         lw      $[[T1:[0-9]+]], 36($fp)
+  ; GP32:         lw      $[[T2:[0-9]+]], 32($fp)
   ; GP32:         xor     $2, $4, $[[T2]]
   ; GP32:         xor     $3, $5, $[[T1]]
-  ; GP32:         lw      $[[T0:[0-9]+]], 24($sp)
+  ; GP32:         lw      $[[T0:[0-9]+]], 40($fp)
   ; GP32:         xor     $4, $6, $[[T0]]
-  ; GP32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; GP32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; GP32:         xor     $5, $7, $[[T3]]
 
   ; GP64:         xor     $2, $4, $6
   ; GP64:         xor     $3, $5, $7
 
-  ; MM32:         lw      $[[T1:[0-9]+]], 20($sp)
-  ; MM32:         lw      $[[T2:[0-9]+]], 16($sp)
+  ; MM32:         lw      $[[T1:[0-9]+]], 36($fp)
+  ; MM32:         lw      $[[T2:[0-9]+]], 32($fp)
   ; MM32:         xor16   $[[T2]], $4
   ; MM32:         xor16   $[[T1]], $5
-  ; MM32:         lw      $[[T0:[0-9]+]], 24($sp)
+  ; MM32:         lw      $[[T0:[0-9]+]], 40($fp)
   ; MM32:         xor16   $[[T0]], $6
-  ; MM32:         lw      $[[T3:[0-9]+]], 28($sp)
+  ; MM32:         lw      $[[T3:[0-9]+]], 44($fp)
   ; MM32:         xor16   $[[T3]], $7
 
   ; MM64:         xor     $2, $4, $6
   ; MM64:         xor     $3, $5, $7
 
   %r = xor i128 %a, %b
   ret i128 %r
 }

test/CodeGen/X86/catchpad-dynamic-alloca.ll

 
 catch.switch:
   %cs = catchswitch within none [label %catch.pad] unwind to caller
 }
 
 ; CHECK-LABEL: $handlerMap$0$test2:
 ; CHECK:      .long   0
 ; CHECK-NEXT: .long   0
-; CHECK-NEXT: .long   8
+; CHECK-NEXT: .long   16

test/CodeGen/X86/libcall-sret.ll

 ; RUN: llc -mtriple=i686-linux-gnu -o - %s | FileCheck %s
 
 @var = global i128 0
 
 ; We were trying to convert the i128 operation into a libcall, but failing to
 ; perform sret demotion when we couldn't return the result in registers. Make
 ; sure we marshal the return properly:
 
 define void @test_sret_libcall(i128 %l, i128 %r) {
 ; CHECK-LABEL: test_sret_libcall:
 
   ; Stack for call: 4(sret ptr), 16(i128 %l), 16(128 %r). So next logical
   ; (aligned) place for the actual sret data is %esp + 20.
-; CHECK: leal 20(%esp), [[SRET_ADDR:%[a-z]+]]
+; CHECK: leal 12(%esp), [[SRET_ADDR:%[a-z]+]]
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl 72(%esp)
 ; CHECK: pushl [[SRET_ADDR]]
 
 ; CHECK: calll __multi3
 
 ; CHECK: addl $44, %esp
-; CHECK-DAG: movl 8(%esp), [[RES0:%[a-z]+]]
-; CHECK-DAG: movl 12(%esp), [[RES1:%[a-z]+]]
-; CHECK-DAG: movl 16(%esp), [[RES2:%[a-z]+]]
-; CHECK-DAG: movl 20(%esp), [[RES3:%[a-z]+]]
+; CHECK-DAG: movl (%esp), [[RES0:%[a-z]+]]
+; CHECK-DAG: movl 4(%esp), [[RES1:%[a-z]+]]
+; CHECK-DAG: movl 8(%esp), [[RES2:%[a-z]+]]
+; CHECK-DAG: movl 12(%esp), [[RES3:%[a-z]+]]
 ; CHECK-DAG: movl [[RES0]], var
 ; CHECK-DAG: movl [[RES1]], var+4
 ; CHECK-DAG: movl [[RES2]], var+8
 ; CHECK-DAG: movl [[RES3]], var+12
   %prod = mul i128 %l, %r
   store i128 %prod, i128* @var
   ret void
 }

test/CodeGen/X86/mul-i1024.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i386-unknown | FileCheck %s --check-prefix=X32
 ; RUN: llc < %s -mtriple=x86_64-unknown | FileCheck %s --check-prefix=X64
 
 define void @test_1024(i1024* %a, i1024* %b, i1024* %out) nounwind {
 ; X32-LABEL: test_1024:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %ebp
 ; X32-NEXT:    movl %esp, %ebp
 ; X32-NEXT:    pushl %ebx
 ; X32-NEXT:    pushl %edi
 ; X32-NEXT:    pushl %esi
-; X32-NEXT:    andl $-8, %esp
-; X32-NEXT:    subl $2496, %esp # imm = 0x9C0
+; X32-NEXT:    andl $-16, %esp
+; X32-NEXT:    subl $2512, %esp # imm = 0x9D0
 ; X32-NEXT:    movl 12(%ebp), %edx
 ; X32-NEXT:    movl 8(%ebp), %ecx
 ; X32-NEXT:    movl 40(%ecx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 44(%ecx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl (%ecx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 4(%ecx), %eax
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 32(%ecx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 36(%ecx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 8(%ecx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 12(%ecx), %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    addl %eax, %ecx
 ; X32-NEXT:    movl %ecx, %edi
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    adcl %ecx, %edx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    adcl $0, %esi
-; X32-NEXT:    movl %esi, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X32-NEXT:    adcl $0, %ebx
 ; X32-NEXT:    addl %eax, %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %ecx, %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %esi
-; X32-NEXT:    addl (%esp), %edi # 4-byte Folded Reload
+; X32-NEXT:    addl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    adcl %ebx, %esi
 ; X32-NEXT:    movl $0, %eax
 ; X32-NEXT:    adcl $0, %eax
 ; X32-NEXT:    movl %eax, %ecx
 ; X32-NEXT:    sbbl %eax, %eax
 ; X32-NEXT:    andl $1, %eax
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    addl %eax, %edi
 ; X32-NEXT:    movl %eax, %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    adcl %eax, %esi
 ; X32-NEXT:    movl %eax, %ebx
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %eax, %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
-; X32-NEXT:    adcl %eax, (%esp) # 4-byte Folded Spill
+; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl %edx, %eax
 ; X32-NEXT:    addl %ecx, %eax
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl %ebx, %ecx
 ; X32-NEXT:    adcl %edx, %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl %ebx, %edx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    addl %edi, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %esi, %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
-; X32-NEXT:    adcl (%esp), %ebx # 4-byte Folded Reload
+; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    addl %eax, %ecx
 ; X32-NEXT:    movl %ecx, %ebx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X32-NEXT:    adcl %ecx, %edi

 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    addl %edx, %ebx
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl %edx, %ebx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
-; X32-NEXT:    movl %esi, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl %esi, %edx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %esi, %edi
 ; X32-NEXT:    addl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    adcl %ecx, %ebx
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %edx, %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
-; X32-NEXT:    movl (%esp), %esi # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
-; X32-NEXT:    movl %esi, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    addl %eax, %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %ecx, %ebx
-; X32-NEXT:    movl (%esp), %ecx # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx # 4-byte Reload
 ; X32-NEXT:    adcl %edi, %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    addl %esi, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    adcl %esi, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    adcl $0, %ebx
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %ecx
-; X32-NEXT:    movl %ecx, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    adcl $0, %edx

 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    adcl $0, %edx
 ; X32-NEXT:    adcl $0, %esi
 ; X32-NEXT:    adcl $0, %ebx
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
-; X32-NEXT:    adcl (%esp), %esi # 4-byte Folded Reload
+; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl $0, %eax
 ; X32-NEXT:    adcl $0, %eax
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl $0, %eax
 ; X32-NEXT:    adcl $0, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl $0, %eax
 ; X32-NEXT:    adcl $0, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    sbbl %eax, %eax
 ; X32-NEXT:    andl $1, %eax

 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
-; X32-NEXT:    adcl (%esp), %edx # 4-byte Folded Reload
+; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill

 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx
 ; X32-NEXT:    adcl $0, %ebx
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    adcl $0, %edx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    adcl $0, %esi
 ; X32-NEXT:    addl %edi, %edx
 ; X32-NEXT:    adcl %ebx, %esi
 ; X32-NEXT:    movl $0, %edi
 ; X32-NEXT:    adcl $0, %edi

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    adcl %ecx, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
-; X32-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    adcl $0, %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl %edx, {{[0-9]+}}(%esp) # 4-byte Folded Spill
-; X32-NEXT:    movl (%esp), %edx # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl %edx, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    adcl $0, %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl $0, %ebx

 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %eax, %ecx
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %eax, %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx
-; X32-NEXT:    movl %ecx, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    addl %ecx, %eax
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx # 4-byte Reload
 ; X32-NEXT:    adcl %edx, %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill

 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %edi, %ecx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %edi, %ebx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl %edi, %esi
-; X32-NEXT:    movl (%esp), %edi # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi # 4-byte Reload
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    addl %edx, %edi
-; X32-NEXT:    movl %edi, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl %ecx, %edx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi # 4-byte Reload
 ; X32-NEXT:    adcl %ebx, %edi
 ; X32-NEXT:    adcl %esi, %eax
 ; X32-NEXT:    movl %eax, %esi
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
-; X32-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl $0, %eax
 ; X32-NEXT:    movl %edx, %ecx
 ; X32-NEXT:    adcl $0, %ecx
 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    adcl $0, %esi
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl $0, %eax
 ; X32-NEXT:    adcl $0, %eax

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax
 ; X32-NEXT:    addl %ebx, %ecx
 ; X32-NEXT:    adcl %edx, %esi
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl %edi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
-; X32-NEXT:    movl (%esp), %edx # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
-; X32-NEXT:    movl %edx, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edi # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    adcl %edx, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
-; X32-NEXT:    movl (%esp), %edx # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %edx # 4-byte Reload
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
-; X32-NEXT:    movl %edx, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edi # 4-byte Folded Reload
 ; X32-NEXT:    movl %edi, %esi
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill

 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill
-; X32-NEXT:    movl (%esp), %eax # 4-byte Reload
+; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl $0, %eax
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ecx # 4-byte Reload
 ; X32-NEXT:    adcl $0, %ecx
 ; X32-NEXT:    movl %esi, %edx
 ; X32-NEXT:    adcl $0, %edx
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl $0, %esi
 ; X32-NEXT:    adcl $0, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl %ebx, %edi
 ; X32-NEXT:    adcl $0, %edi
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl $0, %ebx
 ; X32-NEXT:    adcl $0, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    addl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
-; X32-NEXT:    movl %eax, (%esp) # 4-byte Spill
+; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ecx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %esi # 4-byte Folded Reload
 ; X32-NEXT:    movl %esi, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl %eax, {{[0-9]+}}(%esp) # 4-byte Folded Spill

 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ecx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %edx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
-; X32-NEXT:    addl %esi, (%esp) # 4-byte Folded Spill
+; X32-NEXT:    addl %esi, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl %esi, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl %esi, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl %esi, {{[0-9]+}}(%esp) # 4-byte Folded Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %esi # 4-byte Reload
 ; X32-NEXT:    adcl %esi, {{[0-9]+}}(%esp) # 4-byte Folded Spill

 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    movl %ebx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %ebx # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %ebx # 4-byte Folded Reload
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
-; X32-NEXT:    addl (%esp), %eax # 4-byte Folded Reload
+; X32-NEXT:    addl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload
 ; X32-NEXT:    adcl {{[0-9]+}}(%esp), %eax # 4-byte Folded Reload
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl {{[0-9]+}}(%esp), %eax # 4-byte Reload

test/CodeGen/X86/mul-i512.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i386-unknown | FileCheck %s --check-prefix=X32
 ; RUN: llc < %s -mtriple=x86_64-unknown | FileCheck %s --check-prefix=X64
 
 define void @test_512(i512* %a, i512* %b, i512* %out) nounwind {
 ; X32-LABEL: test_512:
 ; X32:       # BB#0:
 ; X32-NEXT:    pushl %ebp
 ; X32-NEXT:    movl %esp, %ebp
 ; X32-NEXT:    pushl %ebx
 ; X32-NEXT:    pushl %edi
 ; X32-NEXT:    pushl %esi
-; X32-NEXT:    andl $-8, %esp
-; X32-NEXT:    subl $632, %esp # imm = 0x278
+; X32-NEXT:    andl $-16, %esp
+; X32-NEXT:    subl $640, %esp # imm = 0x280
 ; X32-NEXT:    movl 12(%ebp), %ecx
 ; X32-NEXT:    movl 8(%ebp), %ebx
 ; X32-NEXT:    movl (%ebx), %edi
 ; X32-NEXT:    movl 4(%ebx), %eax
 ; X32-NEXT:    movl %eax, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 8(%ebx), %edx
 ; X32-NEXT:    movl %edx, {{[0-9]+}}(%esp) # 4-byte Spill
 ; X32-NEXT:    movl 12(%ebx), %esi

test/CodeGen/X86/osx-private-labels.ll

 
 @private5 = private unnamed_addr constant i64 42
 ; CHECK: .section __TEXT,__literal8,8byte_literals
 ; CHECK-NEXT: .p2align  3
 ; CHECK-NEXT: L_private5:
 
 @private6 = private unnamed_addr constant i128 42
 ; CHECK: .section __TEXT,__literal16,16byte_literals
-; CHECK-NEXT: .p2align  3
+; CHECK-NEXT: .p2align  4
 ; CHECK-NEXT: L_private6:
 
 %struct._objc_class = type { i8* }
 @private7 = private global %struct._objc_class* null, section "__OBJC,__cls_refs,literal_pointers,no_dead_strip"
 ; CHECK: .section __OBJC,__cls_refs,literal_pointers,no_dead_strip
 ; CHECK: .p2align 3
 ; CHECK: L_private7: