[AIX] Extend integer arguments to register width when passed in stack memory.

[cebowleratibm]

Summary

This is a follow up to the previous patch: [AIX] Implement caller arguments passed in stack memory. https://reviews.llvm.org/rGb373ec8ce76a00cd079f70d108b13129c3a4a7b9

This corrects a defect in AIX 64-bit where an i32 is written to the stack with stw (4 bytes) rather than the expected std (8 bytes.) Integer arguments pass on the stack as images of their register representation.

I also took the opportunity to tidy up some of the calling convention AIX tests I added in my last commit. This patch adds the missed assembly expected output for the stack arg int case, which would have caught this problem. I also took the opportunity to reorder some of the calling convention stack arg tests into a more logical order to make them more readable.

[ZarkoCA]

It would be better if the test cleanup was done in a separate patch to make it clearer what the expected output of the patch. But that said, it's not difficult to find added testcase and it would be ok with me to leave the modified test cases.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6821

Suggestion: Move this error to line 6818, just after all the variable declarations required, or make this the last error (i.e. move to line 6832 and have the declarations just above it).

[cebowleratibm]

In D75126#1891669, @ZarkoCA wrote:

It would be better if the test cleanup was done in a separate patch to make it clearer what the expected output of the patch. But that said, it's not difficult to find added testcase and it would be ok with me to leave the modified test cases.

I agree many of the test updates could be a separate NFC patch.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6821

I preferred to keep the errors as a block. I needed to move some basic variable setup above the error block. I can move this error later in the sequence but I wouldn't move the variable setup into the error block, so this wouldn't buy anything. I prefer to keep the current approach.

[cebowleratibm]

Some of the prior test changes have been committed as NFC patches. I've updated the diff and readers can now see the instruction changes for widening arguments to register width before writing to the stack.

[ZarkoCA]

Looks good to me, thanks for updating the tests in a separate patch.

[sfertile]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6821

For 32-bit codegen we legalize i64s by breaking them into 2 i32s. Is i128 handled similarly, or is it a legal type which this error is trying to catch?

6822

Minor nit: word size --> register size, and maybe 'argument' --> 'integer argument'.

[cebowleratibm]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6821

i128 is handled similarly to i64. In PPC64 it will take 2 registers and in PPC32 it will take 4 registers.

The code was not designed to handle this case so I put it in as a defensive measure.

[cebowleratibm]

One comment update. Should be ready to commit.

[sfertile]

LGTM.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

6821

Thanks for checking. In that case I would say make it an assert with a message that reads along the lines: "Illegal integer argument exceeds register size."

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

SDValue PPCTargetLowering::LowerCall_Darwin(
   return FinishCall(CFlags, dl, DAG, RegsToPass, InFlag, Chain, CallSeqStart,
                     Callee, SPDiff, NumBytes, Ins, InVals, CS);
 }
 
 static bool CC_AIX(unsigned ValNo, MVT ValVT, MVT LocVT,
                    CCValAssign::LocInfo LocInfo, ISD::ArgFlagsTy ArgFlags,
                    CCState &State) {
 
+  const PPCSubtarget &Subtarget = static_cast<const PPCSubtarget &>(
+      State.getMachineFunction().getSubtarget());
+  const bool IsPPC64 = Subtarget.isPPC64();
+  const unsigned PtrByteSize = IsPPC64 ? 8 : 4;
+  const MVT RegVT = IsPPC64 ? MVT::i64 : MVT::i32;
+
   if (ValVT == MVT::f128)
     report_fatal_error("f128 is unimplemented on AIX.");
 
+  if (ValVT.isInteger() && (ValVT.getSizeInBits() > RegVT.getSizeInBits()))
+    report_fatal_error("Unexpected argument exceeds word size.");
+
   if (ArgFlags.isByVal())
     report_fatal_error("Passing structure by value is unimplemented.");
 
   if (ArgFlags.isNest())
     report_fatal_error("Nest arguments are unimplemented.");
 
   if (ValVT.isVector() || LocVT.isVector())
     report_fatal_error("Vector arguments are unimplemented on AIX.");
 
-  const PPCSubtarget &Subtarget = static_cast<const PPCSubtarget &>(
-      State.getMachineFunction().getSubtarget());
-  const bool IsPPC64 = Subtarget.isPPC64();
-  const unsigned PtrByteSize = IsPPC64 ? 8 : 4;
-
   static const MCPhysReg GPR_32[] = {// 32-bit registers.
                                      PPC::R3, PPC::R4, PPC::R5, PPC::R6,
                                      PPC::R7, PPC::R8, PPC::R9, PPC::R10};
   static const MCPhysReg GPR_64[] = {// 64-bit registers.
                                      PPC::X3, PPC::X4, PPC::X5, PPC::X6,
                                      PPC::X7, PPC::X8, PPC::X9, PPC::X10};
 
   // Arguments always reserve parameter save area.
   switch (ValVT.SimpleTy) {
   default:
     report_fatal_error("Unhandled value type for argument.");
   case MVT::i64:
     // i64 arguments should have been split to i32 for PPC32.
     assert(IsPPC64 && "PPC32 should have split i64 values.");
     LLVM_FALLTHROUGH;
   case MVT::i1:
   case MVT::i32: {
     const unsigned Offset = State.AllocateStack(PtrByteSize, PtrByteSize);
-    const MVT RegVT = IsPPC64 ? MVT::i64 : MVT::i32;
-    if (unsigned Reg = State.AllocateReg(IsPPC64 ? GPR_64 : GPR_32)) {
-      // Promote integers if needed.
-      if (ValVT.getSizeInBits() < RegVT.getSizeInBits())
-        LocInfo = ArgFlags.isSExt() ? CCValAssign::LocInfo::SExt
-                                    : CCValAssign::LocInfo::ZExt;
+    // AIX integer arguments are always passed in register width.
+    if (unsigned Reg = State.AllocateReg(IsPPC64 ? GPR_64 : GPR_32))
       State.addLoc(CCValAssign::getReg(ValNo, ValVT, Reg, RegVT, LocInfo));
-    }
     else
       State.addLoc(CCValAssign::getMem(ValNo, ValVT, Offset, RegVT, LocInfo));
 
     return false;
   }
   case MVT::f32:
   case MVT::f64: {
     // Parameter save area (PSA) is reserved even if the float passes in fpr.
     const unsigned StoreSize = LocVT.getStoreSize();
     // Floats are always 4-byte aligned in the PSA on AIX.
     // This includes f64 in 64-bit mode for ABI compatibility.
     const unsigned Offset = State.AllocateStack(IsPPC64 ? 8 : StoreSize, 4);
     unsigned FReg = State.AllocateReg(FPR);
     if (FReg)
       State.addLoc(CCValAssign::getReg(ValNo, ValVT, FReg, LocVT, LocInfo));
 
     // Reserve and initialize GPRs or initialize the PSA as required.
-    const MVT RegVT = IsPPC64 ? MVT::i64 : MVT::i32;
     for (unsigned I = 0; I < StoreSize; I += PtrByteSize) {
       if (unsigned Reg = State.AllocateReg(IsPPC64 ? GPR_64 : GPR_32)) {
         assert(FReg && "An FPR should be available when a GPR is reserved.");
         if (State.isVarArg()) {
           // Successfully reserved GPRs are only initialized for vararg calls.
           // Custom handling is required for:
           //   f64 in PPC32 needs to be split into 2 GPRs.
           //   f32 in PPC64 needs to occupy only lower 32 bits of 64-bit GPR.
SDValue PPCTargetLowering::LowerCall_AIX(
   for (unsigned I = 0, E = ArgLocs.size(); I != E;) {
     CCValAssign &VA = ArgLocs[I++];
 
     SDValue Arg = OutVals[VA.getValNo()];
 
     if (!VA.isRegLoc() && !VA.isMemLoc())
       report_fatal_error("Unexpected location for function call argument.");
 
-    if (VA.isRegLoc() && !VA.needsCustom()) {
-      switch (VA.getLocInfo()) {
-      default:
-        report_fatal_error("Unexpected argument extension type.");
-      case CCValAssign::Full:
-        break;
-      case CCValAssign::ZExt:
-        Arg = DAG.getNode(ISD::ZERO_EXTEND, dl, VA.getLocVT(), Arg);
-        break;
-      case CCValAssign::SExt:
-        Arg = DAG.getNode(ISD::SIGN_EXTEND, dl, VA.getLocVT(), Arg);
-        break;
-      }
-      RegsToPass.push_back(std::make_pair(VA.getLocReg(), Arg));
 
+    if (VA.isRegLoc() && !VA.needsCustom()) {
+      RegsToPass.push_back(std::make_pair(VA.getLocReg(), Arg));
       continue;
     }
 
     if (VA.isMemLoc()) {
       SDValue PtrOff =
           DAG.getConstant(VA.getLocMemOffset(), dl, StackPtr.getValueType());
       PtrOff = DAG.getNode(ISD::ADD, dl, PtrVT, StackPtr, PtrOff);
       MemOpChains.push_back(

llvm/test/CodeGen/PowerPC/aix-cc-abi.ll

entry:
   ret void
 }
 
 declare void @test_stackarg_int(i32, i32, i32, i32, i32, i32, i32, i32, i8 zeroext, i16 signext, i32, i64, i32)
 
 ; CHECK-LABEL:     name: call_test_stackarg_int{{.*}}
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
+; 32BIT-DAG:  ADJCALLSTACKDOWN 80, 0, implicit-def dead $r1, implicit $r1
+; 32BIT-DAG:  $r3 = LI 1
+; 32BIT-DAG:  $r4 = LI 2
+; 32BIT-DAG:  $r5 = LI 3
+; 32BIT-DAG:  $r6 = LI 4
+; 32BIT-DAG:  $r7 = LI 5
+; 32BIT-DAG:  $r8 = LI 6
+; 32BIT-DAG:  $r9 = LI 7
+; 32BIT-DAG:  $r10 = LI 8
 ; 32BIT-DAG:  renamable $r[[REGCADDR:[0-9]+]] = LWZtoc @c, $r2 :: (load 4 from got)
 ; 32BIT-DAG:  renamable $r[[REGC:[0-9]+]] = LBZ 0, killed renamable $r[[REGCADDR]] :: (dereferenceable load 1 from @c)
+; 32BIT-DAG:  STW killed renamable $r[[REGC]], 56, $r1 :: (store 4)
 ; 32BIT-DAG:  renamable $r[[REGSIADDR:[0-9]+]] = LWZtoc @si, $r2 :: (load 4 from got)
 ; 32BIT-DAG:  renamable $r[[REGSI:[0-9]+]] = LHA 0, killed renamable $r[[REGSIADDR]] :: (dereferenceable load 2 from @si)
+; 32BIT-DAG:  STW killed renamable $r[[REGSI]], 60, $r1 :: (store 4)
 ; 32BIT-DAG:  renamable $r[[REGIADDR:[0-9]+]] = LWZtoc @i, $r2 :: (load 4 from got)
 ; 32BIT-DAG:  renamable $r[[REGI:[0-9]+]] = LWZ 0, killed renamable $r[[REGIADDR]] :: (dereferenceable load 4 from @i)
+; 32BIT-DAG:  STW killed renamable $r[[REGI]], 64, $r1 :: (store 4)
 ; 32BIT-DAG:  renamable $r[[REGLLIADDR:[0-9]+]] = LWZtoc @lli, $r2 :: (load 4 from got)
 ; 32BIT-DAG:  renamable $r[[REGLLI1:[0-9]+]] = LWZ 0, renamable $r[[REGLLIADDR]] :: (dereferenceable load 4 from @lli, align 8)
-; 32BIT-DAG:  renamable $r[[REGLLI2:[0-9]+]] = LWZ 4, killed renamable $r[[REGLLIADDR]] :: (dereferenceable load 4 from @lli + 4)
-; 32BIT-NEXT: ADJCALLSTACKDOWN 80, 0, implicit-def dead $r1, implicit $r1
-; 32BIT-DAG:  STW killed renamable $r[[REGC]], 56, $r1 :: (store 4)
-; 32BIT-DAG:  STW killed renamable $r[[REGSI]], 60, $r1 :: (store 4)
-; 32BIT-DAG:  STW killed renamable $r[[REGI]], 64, $r1 :: (store 4)
 ; 32BIT-DAG:  STW killed renamable $r[[REGLLI1]], 68, $r1 :: (store 4)
+; 32BIT-DAG:  renamable $r[[REGLLI2:[0-9]+]] = LWZ 4, killed renamable $r[[REGLLIADDR]] :: (dereferenceable load 4 from @lli + 4)
 ; 32BIT-DAG:  STW killed renamable $r[[REGLLI2]], 72, $r1 :: (store 4)
 ; 32BIT-DAG:  STW renamable $r[[REGI]], 76, $r1 :: (store 4)
-; 32BIT-DAG:  $r3 = LI 1
-; 32BIT-DAG:  $r4 = LI 2
-; 32BIT-DAG:  $r5 = LI 3
-; 32BIT-DAG:  $r6 = LI 4
-; 32BIT-DAG:  $r7 = LI 5
-; 32BIT-DAG:  $r8 = LI 6
-; 32BIT-DAG:  $r9 = LI 7
-; 32BIT-DAG:  $r10 = LI 8
 ; 32BIT-NEXT: BL_NOP <mcsymbol .test_stackarg_int>, csr_aix32, implicit-def dead $lr, implicit $rm, implicit $r3, implicit $r4, implicit $r5, implicit $r6, implicit $r7, implicit $r8, implicit $r9, implicit $r10, implicit $r2, implicit-def $r1
 ; 32BIT-NEXT: ADJCALLSTACKUP 80, 0, implicit-def dead $r1, implicit $r1
 
+; CHECKASM-LABEL: .call_test_stackarg_int:
+
+; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
+; ASM32PWR4:       stwu 1, -80(1)
+; ASM32PWR4-DAG:   li 3, 1
+; ASM32PWR4-DAG:   li 4, 2
+; ASM32PWR4-DAG:   li 5, 3
+; ASM32PWR4-DAG:   li 6, 4
+; ASM32PWR4-DAG:   li 7, 5
+; ASM32PWR4-DAG:   li 8, 6
+; ASM32PWR4-DAG:   li 9, 7
+; ASM32PWR4-DAG:   li 10, 8
+; ASM32PWR4-DAG:   lwz [[REGCADDR:[0-9]+]], LC6(2)
+; ASM32PWR4-DAG:   lbz [[REGC:[0-9]+]], 0([[REGCADDR]])
+; ASM32PWR4-DAG:   stw [[REGC]], 56(1)
+; ASM32PWR4-DAG:   lwz [[REGSIADDR:[0-9]+]], LC4(2)
+; ASM32PWR4-DAG:   lha [[REGSI:[0-9]+]], 0([[REGSIADDR]])
+; ASM32PWR4-DAG:   stw [[REGSI]], 60(1)
+; ASM32PWR4-DAG:   lwz [[REGIADDR:[0-9]+]], LC5(2)
+; ASM32PWR4-DAG:   lwz [[REGI:[0-9]+]], 0([[REGIADDR]])
+; ASM32PWR4-DAG:   stw [[REGI]], 64(1)
+; ASM32PWR4-DAG:   lwz [[REGLLIADDR:[0-9]+]], LC7(2)
+; ASM32PWR4-DAG:   lwz [[REGLLI1:[0-9]+]], 0([[REGLLIADDR]])
+; ASM32PWR4-DAG:   stw [[REGLLI1]], 68(1)
+; ASM32PWR4-DAG:   lwz [[REGLLI2:[0-9]+]], 4([[REGLLIADDR]])
+; ASM32PWR4-DAG:   stw [[REGLLI2]], 72(1)
+; ASM32PWR4-DAG:   stw [[REGI]], 76(1)
+; ASM32PWR4-NEXT:  bl .test_stackarg_int
+; ASM32PWR4-NEXT:  nop
+
+; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
+; 64BIT-DAG:   ADJCALLSTACKDOWN 152, 0, implicit-def dead $r1, implicit $r1
+; 64BIT-DAG:   $x3 = LI8 1
+; 64BIT-DAG:   $x4 = LI8 2
+; 64BIT-DAG:   $x5 = LI8 3
+; 64BIT-DAG:   $x6 = LI8 4
+; 64BIT-DAG:   $x7 = LI8 5
+; 64BIT-DAG:   $x8 = LI8 6
+; 64BIT-DAG:   $x9 = LI8 7
+; 64BIT-DAG:   $x10 = LI8 8
+; 64BIT-DAG:   renamable $x[[REGCADDR:[0-9]+]] = LDtoc @c, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $x[[REGC:[0-9]+]] = LBZ8 0, killed renamable $x[[REGCADDR]] :: (dereferenceable load 1 from @c)
+; 64BIT-DAG:   STD killed renamable $x[[REGC]], 112, $x1 :: (store 8)
+; 64BIT-DAG:   renamable $x[[REGSIADDR:[0-9]+]] = LDtoc @si, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $x[[REGSI:[0-9]+]] = LHA8 0, killed renamable $x[[REGSIADDR]] :: (dereferenceable load 2 from @si)
+; 64BIT-DAG:   STD killed renamable $x[[REGSI]], 120, $x1 :: (store 8)
+; 64BIT-DAG:   renamable $x[[REGIADDR:[0-9]+]] = LDtoc @i, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $x[[REGI:[0-9]+]] = LWZ8 0, killed renamable $x[[REGIADDR]] :: (dereferenceable load 4 from @i)
+; 64BIT-DAG:   STD killed renamable $x[[REGI]], 128, $x1 :: (store 8)
+; 64BIT-DAG:   renamable $x[[REGLLIADDR:[0-9]+]] = LDtoc @lli, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $x[[REGLLI:[0-9]+]] = LD 0, killed renamable $x[[REGLLIADDR]] :: (dereferenceable load 8 from @lli)
+; 64BIT-DAG:   STD killed renamable $x[[REGLLI]], 136, $x1 :: (store 8)
+; 64BIT-DAG:   STD renamable $x[[REGI]], 144, $x1 :: (store 8)
+; 64BIT-NEXT:  BL8_NOP <mcsymbol .test_stackarg_int>, csr_aix64, implicit-def dead $lr8, implicit $rm, implicit $x3, implicit $x4, implicit $x5, implicit $x6, implicit $x7, implicit $x8, implicit $x9, implicit $x10, implicit $x2, implicit-def $r1
+; 64BIT-NEXT:  ADJCALLSTACKUP 152, 0, implicit-def dead $r1, implicit $r1
+
+; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
+; ASM64PWR4-DAG:   stdu 1, -160(1)
+; ASM64PWR4-DAG:   li 3, 1
+; ASM64PWR4-DAG:   li 4, 2
+; ASM64PWR4-DAG:   li 5, 3
+; ASM64PWR4-DAG:   li 6, 4
+; ASM64PWR4-DAG:   li 7, 5
+; ASM64PWR4-DAG:   li 8, 6
+; ASM64PWR4-DAG:   li 9, 7
+; ASM64PWR4-DAG:   li 10, 8
+; ASM64PWR4-DAG:   ld [[REGCADDR:[0-9]+]], LC5(2)
+; ASM64PWR4-DAG:   lbz [[REGC:[0-9]+]], 0([[REGCADDR]])
+; ASM64PWR4-DAG:   std [[REGC]], 112(1)
+; ASM64PWR4-DAG:   ld [[REGSIADDR:[0-9]+]], LC3(2)
+; ASM64PWR4-DAG:   lha [[REGSI:[0-9]+]], 0([[REGSIADDR]])
+; ASM64PWR4-DAG:   std [[REGSI]], 120(1)
+; ASM64PWR4-DAG:   ld [[REGIADDR:[0-9]+]], LC4(2)
+; ASM64PWR4-DAG:   lwz [[REGI:[0-9]+]], 0([[REGIADDR]])
+; ASM64PWR4-DAG:   std [[REGI]], 128(1)
+; ASM64PWR4-DAG:   ld [[REGLLIADDR:[0-9]+]], LC6(2)
+; ASM64PWR4-DAG:   ld [[REGLLI:[0-9]+]], 0([[REGLLIADDR]])
+; ASM64PWR4-DAG:   std [[REGLLI]], 136(1)
+; ASM64PWR4-DAG:   std [[REGI]], 144(1)
+; ASM64PWR4-NEXT:  bl .test_stackarg_int
+; ASM64PWR4-NEXT:  nop
+
 ; Basic saving of floating point type arguments to the parameter save area.
 ; The float and double arguments will pass in both fpr as well as parameter save area.
 define void @call_test_stackarg_float() {
 entry:
   %0 = load float, float* @f, align 4
   %1 = load double, double* @d, align 8
   call void @test_stackarg_float(i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, float %0, double %1)
   ret void
 }
 
 declare void @test_stackarg_float(i32, i32, i32, i32, i32, i32, i32, i32, float, double)
 
 ; CHECK-LABEL:     name:            call_test_stackarg_float
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
-; 32BIT-DAG:   renamable $r[[REGF:[0-9]+]] = LWZtoc @f, $r2 :: (load 4 from got)
-; 32BIT-DAG:   renamable $f1 = LFS 0, killed renamable $r[[REGF]] :: (dereferenceable load 4 from @f)
-; 32BIT-DAG:   renamable $r[[REGD:[0-9]+]] = LWZtoc @d, $r2 :: (load 4 from got)
-; 32BIT-DAG:   renamable $f2 = LFD 0, killed renamable $r[[REGD]] :: (dereferenceable load 8 from @d)
-; 32BIT-NEXT:  ADJCALLSTACKDOWN 68, 0, implicit-def dead $r1, implicit $r1
-; 32BIT-DAG:   STFS renamable $f1, 56, $r1 :: (store 4)
-; 32BIT-DAG:   STFD renamable $f2, 60, $r1 :: (store 8)
+; 32BIT-DAG:   ADJCALLSTACKDOWN 68, 0, implicit-def dead $r1, implicit $r1
 ; 32BIT-DAG:   $r3 = LI 1
 ; 32BIT-DAG:   $r4 = LI 2
 ; 32BIT-DAG:   $r5 = LI 3
 ; 32BIT-DAG:   $r6 = LI 4
 ; 32BIT-DAG:   $r7 = LI 5
 ; 32BIT-DAG:   $r8 = LI 6
 ; 32BIT-DAG:   $r9 = LI 7
 ; 32BIT-DAG:   $r10 = LI 8
+; 32BIT-DAG:   renamable $r[[REGF:[0-9]+]] = LWZtoc @f, $r2 :: (load 4 from got)
+; 32BIT-DAG:   renamable $f1 = LFS 0, killed renamable $r[[REGF]] :: (dereferenceable load 4 from @f)
+; 32BIT-DAG:   renamable $r[[REGD:[0-9]+]] = LWZtoc @d, $r2 :: (load 4 from got)
+; 32BIT-DAG:   renamable $f2 = LFD 0, killed renamable $r[[REGD]] :: (dereferenceable load 8 from @d)
+; 32BIT-DAG:   STFS renamable $f1, 56, $r1 :: (store 4)
+; 32BIT-DAG:   STFD renamable $f2, 60, $r1 :: (store 8)
 ; 32BIT-NEXT:  BL_NOP <mcsymbol .test_stackarg_float>, csr_aix32, implicit-def dead $lr, implicit $rm, implicit $r3, implicit $r4, implicit killed $r5, implicit killed $r6, implicit killed $r7, implicit killed $r8, implicit killed $r9, implicit killed $r10, implicit $f1, implicit $f2, implicit $r2, implicit-def $r1
 ; 32BIT-NEXT:  ADJCALLSTACKUP 68, 0, implicit-def dead $r1, implicit $r1
 
 ; CHECKASM-LABEL: .call_test_stackarg_float:
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; ASM32PWR4:      stwu 1, -80(1)
-; ASM32PWR4-DAG:  lwz [[REGF:[0-9]+]], LC8(2)
-; ASM32PWR4-DAG:  lfs 1, 0([[REGF]])
-; ASM32PWR4-DAG:  lwz [[REGD:[0-9]+]], LC9(2)
-; ASM32PWR4-DAG:  lfd 2, 0([[REGD:[0-9]+]])
-; ASM32PWR4-DAG:  stfs 1, 56(1)
-; ASM32PWR4-DAG:  stfd 2, 60(1)
 ; ASM32PWR4-DAG:  li 3, 1
 ; ASM32PWR4-DAG:  li 4, 2
 ; ASM32PWR4-DAG:  li 5, 3
 ; ASM32PWR4-DAG:  li 6, 4
 ; ASM32PWR4-DAG:  li 7, 5
 ; ASM32PWR4-DAG:  li 8, 6
 ; ASM32PWR4-DAG:  li 9, 7
 ; ASM32PWR4-DAG:  li 10, 8
+; ASM32PWR4-DAG:  lwz [[REGF:[0-9]+]], LC8(2)
+; ASM32PWR4-DAG:  lfs 1, 0([[REGF]])
+; ASM32PWR4-DAG:  lwz [[REGD:[0-9]+]], LC9(2)
+; ASM32PWR4-DAG:  lfd 2, 0([[REGD:[0-9]+]])
+; ASM32PWR4-DAG:  stfs 1, 56(1)
+; ASM32PWR4-DAG:  stfd 2, 60(1)
 ; ASM32PWR4-NEXT: bl .test_stackarg_float
 ; ASM32PWR4-NEXT: nop
 ; ASM32PWR4-NEXT: addi 1, 1, 80
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
-; 64BIT-DAG:   renamable $x[[REGF:[0-9]+]] = LDtoc @f, $x2 :: (load 8 from got)
-; 64BIT-DAG:   renamable $f1 = LFS 0, killed renamable $x[[REGF]] :: (dereferenceable load 4 from @f)
-; 64BIT-DAG:   renamable $x[[REGD:[0-9]+]] = LDtoc @d, $x2 :: (load 8 from got)
-; 64BIT-DAG:   renamable $f2 = LFD 0, killed renamable $x[[REGD]] :: (dereferenceable load 8 from @d)
-; 64BIT-NEXT:  ADJCALLSTACKDOWN 128, 0, implicit-def dead $r1, implicit $r1
-; 64BIT-DAG:   STFS renamable $f1, 112, $x1 :: (store 4)
-; 64BIT-DAG:   STFD renamable $f2, 120, $x1 :: (store 8)
+; 64BIT-DAG:   ADJCALLSTACKDOWN 128, 0, implicit-def dead $r1, implicit $r1
 ; 64BIT-DAG:   $x3 = LI8 1
 ; 64BIT-DAG:   $x4 = LI8 2
 ; 64BIT-DAG:   $x5 = LI8 3
 ; 64BIT-DAG:   $x6 = LI8 4
 ; 64BIT-DAG:   $x7 = LI8 5
 ; 64BIT-DAG:   $x8 = LI8 6
 ; 64BIT-DAG:   $x9 = LI8 7
 ; 64BIT-DAG:   $x10 = LI8 8
+; 64BIT-DAG:   renamable $x[[REGF:[0-9]+]] = LDtoc @f, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $f1 = LFS 0, killed renamable $x[[REGF]] :: (dereferenceable load 4 from @f)
+; 64BIT-DAG:   renamable $x[[REGD:[0-9]+]] = LDtoc @d, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $f2 = LFD 0, killed renamable $x[[REGD]] :: (dereferenceable load 8 from @d)
+; 64BIT-DAG:   STFS renamable $f1, 112, $x1 :: (store 4)
+; 64BIT-DAG:   STFD renamable $f2, 120, $x1 :: (store 8)
 ; 64BIT-NEXT:  BL8_NOP <mcsymbol .test_stackarg_float>, csr_aix64, implicit-def dead $lr8, implicit $rm, implicit $x3, implicit $x4, implicit killed $x5, implicit killed $x6, implicit killed $x7, implicit killed $x8, implicit killed $x9, implicit killed $x10, implicit $f1, implicit $f2, implicit $x2, implicit-def $r1
 ; 64BIT-NEXT:  ADJCALLSTACKUP 128, 0, implicit-def dead $r1, implicit $r1
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; ASM64PWR4:      stdu 1, -128(1)
-; ASM64PWR4-DAG:  ld [[REGF:[0-9]+]], LC7(2)
-; ASM64PWR4-DAG:  lfs 1, 0([[REGF]])
-; ASM64PWR4-DAG:  ld [[REGD:[0-9]+]], LC8(2)
-; ASM64PWR4-DAG:  lfd 2, 0([[REGD]])
-; ASM64PWR4-DAG:  stfs 1, 112(1)
-; ASM64PWR4-DAG:  stfd 2, 120(1)
 ; ASM64PWR4-DAG:  li 3, 1
 ; ASM64PWR4-DAG:  li 4, 2
 ; ASM64PWR4-DAG:  li 5, 3
 ; ASM64PWR4-DAG:  li 6, 4
 ; ASM64PWR4-DAG:  li 7, 5
 ; ASM64PWR4-DAG:  li 8, 6
 ; ASM64PWR4-DAG:  li 9, 7
 ; ASM64PWR4-DAG:  li 10, 8
+; ASM64PWR4-DAG:  ld [[REGF:[0-9]+]], LC7(2)
+; ASM64PWR4-DAG:  lfs 1, 0([[REGF]])
+; ASM64PWR4-DAG:  ld [[REGD:[0-9]+]], LC8(2)
+; ASM64PWR4-DAG:  lfd 2, 0([[REGD]])
+; ASM64PWR4-DAG:  stfs 1, 112(1)
+; ASM64PWR4-DAG:  stfd 2, 120(1)
 ; ASM64PWR4-NEXT: bl .test_stackarg_float
 ; ASM64PWR4-NEXT: nop
 ; ASM64PWR4-NEXT: addi 1, 1, 128
 
 define void @call_test_stackarg_float2() {
 entry:
   %0 = load double, double* @d, align 8
   call void (i32, i32, i32, i32, i32, i32, ...) @test_stackarg_float2(i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, double %0)
   ret void
 }
 
 declare void @test_stackarg_float2(i32, i32, i32, i32, i32, i32, ...)
 
 ; CHECK-LABEL:     name: call_test_stackarg_float2{{.*}}
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
-; 32BIT-DAG:   renamable $r[[REG:[0-9]+]] = LWZtoc @d, $r2 :: (load 4 from got)
-; 32BIT-DAG:   renamable $f1 = LFD 0, killed renamable $r[[REG]] :: (dereferenceable load 8 from @d)
 ; 32BIT-DAG:   ADJCALLSTACKDOWN 56, 0, implicit-def dead $r1, implicit $r1
 ; 32BIT-DAG:   $r3 = LI 1
 ; 32BIT-DAG:   $r4 = LI 2
 ; 32BIT-DAG:   $r5 = LI 3
 ; 32BIT-DAG:   $r6 = LI 4
 ; 32BIT-DAG:   $r7 = LI 5
 ; 32BIT-DAG:   $r8 = LI 6
+; 32BIT-DAG:   renamable $r[[REG:[0-9]+]] = LWZtoc @d, $r2 :: (load 4 from got)
+; 32BIT-DAG:   renamable $f1 = LFD 0, killed renamable $r[[REG]] :: (dereferenceable load 8 from @d)
 ; 32BIT-DAG:   STFD renamable $f1, 0, %stack.0 :: (store 8 into %stack.0)
 ; 32BIT-DAG:   renamable $r9 = LWZ 0, %stack.0 :: (load 4 from %stack.0, align 8)
 ; 32BIT-DAG:   renamable $r10 = LWZ 4, %stack.0 :: (load 4 from %stack.0 + 4)
 ; 32BIT-NEXT:   BL_NOP <mcsymbol .test_stackarg_float2>, csr_aix32, implicit-def dead $lr, implicit $rm, implicit $r3, implicit killed $r4, implicit killed $r5, implicit killed $r6, implicit killed $r7, implicit killed $r8, implicit $f1, implicit $r9, implicit $r10, implicit $r2, implicit-def $r1
 ; 32BIT-NEXT:   ADJCALLSTACKUP 56, 0, implicit-def dead $r1, implicit $r1
 
 ; CHECKASM-LABEL: .call_test_stackarg_float2:
 

 ; ASM32PWR4-DAG: stfd 1, 56(1)
 ; ASM32PWR4-DAG: lwz 9, 56(1)
 ; ASM32PWR4-DAG: lwz 10, 60(1)
 ; ASM32PWR4-NEXT: bl .test_stackarg_float2
 ; ASM32PWR4-NEXT: nop
 ; ASM32PWR4-NEXT: addi 1, 1, 64
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
-; 64BIT-DAG:   renamable $x[[REG:[0-9]+]] = LDtoc @d, $x2 :: (load 8 from got)
-; 64BIT-DAG:   renamable $f1 = LFD 0, killed renamable $x[[REG]] :: (dereferenceable load 8 from @d)
 ; 64BIT-DAG:   ADJCALLSTACKDOWN 112, 0, implicit-def dead $r1, implicit $r1
 ; 64BIT-DAG:   $x3 = LI8 1
 ; 64BIT-DAG:   $x4 = LI8 2
 ; 64BIT-DAG:   $x5 = LI8 3
 ; 64BIT-DAG:   $x6 = LI8 4
 ; 64BIT-DAG:   $x7 = LI8 5
 ; 64BIT-DAG:   $x8 = LI8 6
+; 64BIT-DAG:   renamable $x[[REG:[0-9]+]] = LDtoc @d, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $f1 = LFD 0, killed renamable $x[[REG]] :: (dereferenceable load 8 from @d)
 ; 64BIT-DAG:   STFD renamable $f1, 0, %stack.0 :: (store 8 into %stack.0)
 ; 64BIT-DAG:   renamable $x9 = LD 0, %stack.0 :: (load 8 from %stack.0)
 ; 64BIT-NEXT:  BL8_NOP <mcsymbol .test_stackarg_float2>, csr_aix64, implicit-def dead $lr8, implicit $rm, implicit $x3, implicit killed $x4, implicit killed $x5, implicit killed $x6, implicit killed $x7, implicit killed $x8, implicit $f1, implicit $x9, implicit $x2, implicit-def $r1
 ; 64BIT-NEXT:  ADJCALLSTACKUP 112, 0, implicit-def dead $r1, implicit $r1
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; ASM64PWR4:     stdu 1, -128(1)
 ; ASM64PWR4-DAG: li 3, 1

 }
 
 declare void @test_stackarg_float3(i32, i32, i32, i32, i32, i32, i32, ...)
 
 ; CHECK-LABEL:     name: call_test_stackarg_float3{{.*}}
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; In 32-bit the double arg is written to memory because it cannot be fully stored in the last 32-bit GPR.
-; 32BIT-DAG:   renamable $r[[REGD:[0-9]+]] = LWZtoc @d, $r2 :: (load 4 from got)
-; 32BIT-DAG:   renamable $f1 = LFD 0, killed renamable $r[[REGD]] :: (dereferenceable load 8 from @d)
-; 32BIT-DAG:   renamable $r[[REGF:[0-9]+]] = LWZtoc @f, $r2 :: (load 4 from got)
-; 32BIT-DAG:   renamable $f2 = LFS 0, killed renamable $r[[REGF]] :: (dereferenceable load 4 from @f)
 ; 32BIT-DAG:   ADJCALLSTACKDOWN 64, 0, implicit-def dead $r1, implicit $r1
-; 32BIT-DAG:   STFD renamable $f1, 52, $r1 :: (store 8)
-; 32BIT-DAG:   STFS renamable $f2, 60, $r1 :: (store 4)
 ; 32BIT-DAG:   $r3 = LI 1
 ; 32BIT-DAG:   $r4 = LI 2
 ; 32BIT-DAG:   $r5 = LI 3
 ; 32BIT-DAG:   $r6 = LI 4
 ; 32BIT-DAG:   $r7 = LI 5
 ; 32BIT-DAG:   $r8 = LI 6
 ; 32BIT-DAG:   $r9 = LI 7
+; 32BIT-DAG:   renamable $r[[REGD:[0-9]+]] = LWZtoc @d, $r2 :: (load 4 from got)
+; 32BIT-DAG:   renamable $f1 = LFD 0, killed renamable $r[[REGD]] :: (dereferenceable load 8 from @d)
+; 32BIT-DAG:   renamable $r[[REGF:[0-9]+]] = LWZtoc @f, $r2 :: (load 4 from got)
+; 32BIT-DAG:   renamable $f2 = LFS 0, killed renamable $r[[REGF]] :: (dereferenceable load 4 from @f)
+; 32BIT-DAG:   STFD renamable $f1, 52, $r1 :: (store 8)
+; 32BIT-DAG:   STFS renamable $f2, 60, $r1 :: (store 4)
 ; 32BIT-DAG:   STFD renamable $f1, 0, %stack.0 :: (store 8 into %stack.0)
 ; 32BIT-DAG:   renamable $r10 = LWZ 0, %stack.0 :: (load 4 from %stack.0, align 8)
 ; 32BIT-NEXT:  BL_NOP <mcsymbol .test_stackarg_float3>, csr_aix32, implicit-def dead $lr, implicit $rm, implicit $r3, implicit killed $r4, implicit killed $r5, implicit killed $r6, implicit killed $r7, implicit killed $r8, implicit killed $r9, implicit $f1, implicit $r10, implicit $f2, implicit $r2, implicit-def $r1
 ; 32BIT-NEXT:  ADJCALLSTACKUP 64, 0, implicit-def dead $r1, implicit $r1
 
 ; CHECKASM-LABEL: .call_test_stackarg_float3:
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; ASM32PWR4:       stwu 1, -80(1)
-; ASM32PWR4-DAG:   lwz [[REGD:[0-9]+]], LC9(2)
-; ASM32PWR4-DAG:   lfd 1, 0([[REGD]])
-; ASM32PWR4-DAG:   lwz [[REGF:[0-9]+]], LC8(2)
-; ASM32PWR4-DAG:   lfs 2, 0([[REGF]])
-; ASM32PWR4-DAG:   stfd 1, 52(1)
-; ASM32PWR4-DAG:   stfs 2, 60(1)
 ; ASM32PWR4-DAG:   li 3, 1
 ; ASM32PWR4-DAG:   li 4, 2
 ; ASM32PWR4-DAG:   li 5, 3
 ; ASM32PWR4-DAG:   li 6, 4
 ; ASM32PWR4-DAG:   li 7, 5
 ; ASM32PWR4-DAG:   li 8, 6
 ; ASM32PWR4-DAG:   li 9, 7
+; ASM32PWR4-DAG:   lwz [[REGD:[0-9]+]], LC9(2)
+; ASM32PWR4-DAG:   lfd 1, 0([[REGD]])
+; ASM32PWR4-DAG:   lwz [[REGF:[0-9]+]], LC8(2)
+; ASM32PWR4-DAG:   lfs 2, 0([[REGF]])
+; ASM32PWR4-DAG:   stfd 1, 52(1)
+; ASM32PWR4-DAG:   stfs 2, 60(1)
 ; ASM32PWR4-DAG:   stfd 1, 72(1)
 ; ASM32PWR4-DAG:   lwz 10, 72(1)
 ; ASM32PWR4-NEXT:  bl .test_stackarg_float3
 ; ASM32PWR4-NEXT:  nop
 ; ASM32PWR4-NEXT:  addi 1, 1, 80
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; In 64-bit the double arg is not written to memory because it is fully stored in the last 64-bit GPR.
-; 64BIT-DAG:   renamable $x[[REGD:[0-9]+]] = LDtoc @d, $x2 :: (load 8 from got)
-; 64BIT-DAG:   renamable $f1 = LFD 0, killed renamable $x[[REGD]] :: (dereferenceable load 8 from @d)
-; 64BIT-DAG:   renamable $x[[REGF:[0-9]+]] = LDtoc @f, $x2 :: (load 8 from got)
-; 64BIT-DAG:   renamable $f2 = LFS 0, killed renamable $x[[REGF]] :: (dereferenceable load 4 from @f)
 ; 64BIT-DAG:   ADJCALLSTACKDOWN 120, 0, implicit-def dead $r1, implicit $r1
-; 64BIT-DAG:   STFS renamable $f2, 112, $x1 :: (store 4)
 ; 64BIT-DAG:   $x3 = LI8 1
 ; 64BIT-DAG:   $x4 = LI8 2
 ; 64BIT-DAG:   $x5 = LI8 3
 ; 64BIT-DAG:   $x6 = LI8 4
 ; 64BIT-DAG:   $x7 = LI8 5
 ; 64BIT-DAG:   $x8 = LI8 6
 ; 64BIT-DAG:   $x9 = LI8 7
+; 64BIT-DAG:   renamable $x[[REGD:[0-9]+]] = LDtoc @d, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $f1 = LFD 0, killed renamable $x[[REGD]] :: (dereferenceable load 8 from @d)
+; 64BIT-DAG:   renamable $x[[REGF:[0-9]+]] = LDtoc @f, $x2 :: (load 8 from got)
+; 64BIT-DAG:   renamable $f2 = LFS 0, killed renamable $x[[REGF]] :: (dereferenceable load 4 from @f)
+; 64BIT-DAG:   STFS renamable $f2, 112, $x1 :: (store 4)
 ; 64BIT-DAG:   STFD renamable $f1, 0, %stack.0 :: (store 8 into %stack.0)
 ; 64BIT-DAG:   renamable $x10 = LD 0, %stack.0 :: (load 8 from %stack.0)
 ; 64BIT-NEXT:  BL8_NOP <mcsymbol .test_stackarg_float3>, csr_aix64, implicit-def dead $lr8, implicit $rm, implicit $x3, implicit killed $x4, implicit killed $x5, implicit killed $x6, implicit killed $x7, implicit killed $x8, implicit killed $x9, implicit $f1, implicit $x10, implicit $f2, implicit $x2, implicit-def $r1
 
 ; 64BIT-NEXT: ADJCALLSTACKUP 120, 0, implicit-def dead $r1, implicit $r1
 
 ; The DAG block permits some invalid inputs for the benefit of allowing more valid orderings.
 ; ASM64PWR4:       stdu 1, -128(1)
-; ASM64PWR4-DAG:   ld [[REGD:[0-9]+]], LC8(2)
-; ASM64PWR4-DAG:   lfd 1, 0([[REGD]])
-; ASM64PWR4-DAG:   ld [[REGF:[0-9]+]], LC7(2)
-; ASM64PWR4-DAG:   lfs 2, 0([[REGF]])
-; ASM64PWR4-DAG:   stfs 2, 112(1)
 ; ASM64PWR4-DAG:   li 3, 1
 ; ASM64PWR4-DAG:   li 4, 2
 ; ASM64PWR4-DAG:   li 5, 3
 ; ASM64PWR4-DAG:   li 6, 4
 ; ASM64PWR4-DAG:   li 7, 5
 ; ASM64PWR4-DAG:   li 8, 6
 ; ASM64PWR4-DAG:   li 9, 7
+; ASM64PWR4-DAG:   ld [[REGD:[0-9]+]], LC8(2)
+; ASM64PWR4-DAG:   lfd 1, 0([[REGD]])
+; ASM64PWR4-DAG:   ld [[REGF:[0-9]+]], LC7(2)
+; ASM64PWR4-DAG:   lfs 2, 0([[REGF]])
+; ASM64PWR4-DAG:   stfs 2, 112(1)
 ; ASM64PWR4-DAG:   stfd 1, 120(1)
 ; ASM64PWR4-DAG:   ld 10, 120(1)
 ; ASM64PWR4-NEXT:  bl .test_stackarg_float3
 ; ASM64PWR4-NEXT:  nop
 ; ASM64PWR4-NEXT:  addi 1, 1, 128