[PowerPC] Replace MFVSRLD with MFVSRD when the vector is symmetrical

[stefanp]

Summary

The MFVSRD is faster than the MFVSRLD instruction and if the input vector is
symmetrical then both instructions produce the same result and we should prefer
the faster one.

This patch mainly looks at symmetrical situations that are known to arise after
a vector doubleword swap.

[shchenz]

Can we fix this at the place where MFVSRLD is generated? (in DAG-ISEL?)

[nemanjai]

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

411

Is this just any instruction that sets isCommutable = 1?

[amyk]

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

411

Could we maybe add a comment as to why we are looking at these instructions specifically?

I noticed that Nemanja is mostly correct in that a majority of these set isCommutable = 1, but with the exception of VMULLD, VMULHSD, VMULHUD.

[stefanp]

Updated the patch to use isCommutable instead of listing the instructions
individually.

[stefanp]

In D114800#3163072, @shchenz wrote:

Can we fix this at the place where MFVSRLD is generated? (in DAG-ISEL?)

I see what you mean. This can be done in DAG-ISEL.
However, there are a couple of reasons why I chose to do this here.

1. I think it is easier to do here and the code is simpler to understand. We are looking to replace MFVSRLD so we just look for that instruction and then replace it. In DAG ISEL we would have to first figure out which extract element node eventually gets turned into this instruction and then see if it can be replaced.
2. There are situations where the choice of preceding instructions matters to whether or not we can use the other doubleword. Again, it would be a situation where we would have to figure out what a node is turned into before deciding if the doubleword is symmetrical.

[amyk]

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

408

Maybe we can try an early exit if this works:

if (!DefVecReg->getDesc().isCommutable())
  return false;

[stefanp]

Added an early exit.

[qiucf]

In D114800#3197765, @stefanp wrote:

In D114800#3163072, @shchenz wrote:

Can we fix this at the place where MFVSRLD is generated? (in DAG-ISEL?)

I see what you mean. This can be done in DAG-ISEL.
However, there are a couple of reasons why I chose to do this here.

1. I think it is easier to do here and the code is simpler to understand. We are looking to replace MFVSRLD so we just look for that instruction and then replace it. In DAG ISEL we would have to first figure out which extract element node eventually gets turned into this instruction and then see if it can be replaced.
2. There are situations where the choice of preceding instructions matters to whether or not we can use the other doubleword. Again, it would be a situation where we would have to figure out what a node is turned into before deciding if the doubleword is symmetrical.

Will putting code in post-isel peephole help? See D97658.

[stefanp]

In D114800#3253393, @qiucf wrote:

In D114800#3197765, @stefanp wrote:

In D114800#3163072, @shchenz wrote:

Can we fix this at the place where MFVSRLD is generated? (in DAG-ISEL?)

I see what you mean. This can be done in DAG-ISEL.
However, there are a couple of reasons why I chose to do this here.

1. I think it is easier to do here and the code is simpler to understand. We are looking to replace MFVSRLD so we just look for that instruction and then replace it. In DAG ISEL we would have to first figure out which extract element node eventually gets turned into this instruction and then see if it can be replaced.
2. There are situations where the choice of preceding instructions matters to whether or not we can use the other doubleword. Again, it would be a situation where we would have to figure out what a node is turned into before deciding if the doubleword is symmetrical.

Will putting code in post-isel peephole help? See D97658.

I think I see what you mean based on that patch. I could try to move it there and re-use isVSXSwap. I will try to move the code there and see what I get.

[nemanjai]

Something to consider when it comes to deciding where to do something like this is that any work we do on the SDAG:

1. Will be basic-block local obviously
2. Will not be useful in the future when we switch to GISel

[stefanp]

Added a test case to show the issue with basic block local.

[stefanp]

In D114800#3254289, @stefanp wrote:

In D114800#3253393, @qiucf wrote:

In D114800#3197765, @stefanp wrote:

In D114800#3163072, @shchenz wrote:

Can we fix this at the place where MFVSRLD is generated? (in DAG-ISEL?)

I see what you mean. This can be done in DAG-ISEL.
However, there are a couple of reasons why I chose to do this here.

1. I think it is easier to do here and the code is simpler to understand. We are looking to replace MFVSRLD so we just look for that instruction and then replace it. In DAG ISEL we would have to first figure out which extract element node eventually gets turned into this instruction and then see if it can be replaced.
2. There are situations where the choice of preceding instructions matters to whether or not we can use the other doubleword. Again, it would be a situation where we would have to figure out what a node is turned into before deciding if the doubleword is symmetrical.

Will putting code in post-isel peephole help? See D97658.

I think I see what you mean based on that patch. I could try to move it there and re-use isVSXSwap. I will try to move the code there and see what I get.

So, I ended up also fully implementing this in post-isel peephole and then I compared the implementations. The conclusion I came to is that I would prefer to leave the implementation here in the MIPeephole.

Now, here is the long answer.
I took Nemanja's comment into consideration:

Something to consider when it comes to deciding where to do something like this is that any work we do on the SDAG:

1. Will be basic-block local obviously
2. Will not be useful in the future when we switch to GISel

For point number 1. the basic-block local aspect of the ISel actually causes the ISel implementation to miss opportunities. To demonstrate this I added this test: ppc64-mfvsrld-removal.ll.

If you look at that test in ISel the Basic Block that has the MFVSRLD instruction looks like this:

SelectionDAG has 8 nodes:
  t0: ch = EntryToken
        t2: v2i64,ch = CopyFromReg t0, Register:v2i64 %0
      t4: i64 = MFVSRLD t2
    t6: ch = CopyToReg t0, Register:i64 %1, t4
  t8: ch = B BasicBlock:ch<cleanup 0x10024b22b10>, t6

There isn't enough history in that BB to allow us to transform the instruction.
On the other hand, in MIPeephole we can see where the values are coming from:

# Machine code for function getVecSplit: IsSSA, TracksLiveness
Function Live Ins: $x3 in %4, $v2 in %5, $v3 in %6

bb.0.entry:
  successors: %bb.2(0x30000000), %bb.1(0x50000000); %bb.2(37.50%), %bb.1(62.50%)
  liveins: $x3, $v2, $v3
  %6:vrrc = COPY $v3
  %5:vrrc = COPY $v2
  %4:g8rc = COPY $x3
  %7:gprc = COPY %4.sub_32:g8rc
  %8:vrrc = XXPERMDI %5:vrrc, %5:vrrc, 2
  %0:vrrc = VADDUDM killed %8:vrrc, %5:vrrc     <---- Comes from here
  %9:crrc = CMPLWI killed %7:gprc, 0
  BCC 76, killed %9:crrc, %bb.2
  B %bb.1

bb.1.if.then:
; predecessors: %bb.0
  successors: %bb.3(0x80000000); %bb.3(100.00%)

  %1:g8rc = MFVSRLD %0:vrrc
  B %bb.3

bb.2.if.else:
; predecessors: %bb.0
  successors: %bb.3(0x80000000); %bb.3(100.00%)

  %10:vrrc = VADDUDM %0:vrrc, %6:vrrc
  ADJCALLSTACKDOWN 32, 0, implicit-def dead $r1, implicit $r1
  $v2 = COPY %10:vrrc
  BL8_NOP @callee, <regmask $cr2 $cr3 $cr4 $f14 $f15 $f16 $f17 $f18 $f19 $f20 $f21 $f22 $f23 $f24 $f25 $f26 $f27 $f28 $f29 $f30 $f31 $r14 $r15 $r16 $r17 $r18 $r19 $r20 $r21 $r22 $r23 $r24 $r25 and 60 more...>, implicit-def dead $lr8, implicit $rm, implicit $v2, implicit $x2, implicit-def $r1, implicit-def $x3
  ADJCALLSTACKUP 32, 0, implicit-def dead $r1, implicit $r1
  %11:g8rc_and_g8rc_nox0 = COPY $x3
  %2:g8rc = nsw ADDI8 %11:g8rc_and_g8rc_nox0, 42

bb.3.cleanup:
; predecessors: %bb.1, %bb.2

  %3:g8rc = PHI %1:g8rc, %bb.1, %2:g8rc, %bb.2
  $x3 = COPY %3:g8rc
  BLR8 implicit $lr8, implicit $rm, implicit $x3

# End machine code for function getVecSplit.

Therefore for this test we only catch the opportunity if we check for it in MIPeephole.

Also there are other reasons why I wanted to keep the patch as-is:

1. As Nemanja mentioned we may be moving to GISel and so this work would have to be re-done in that situation if we did it in ISel.
2. In ISel there are a number of nodes that we have to "look though" like COPY_TO_REGCLASS that makes the code more complicated. In MIPeephole a lot of those nodes have now been simplified away.
3. I didn't see any advantages to doing this earlier as I don't think that this transformation will make much of a difference to passes that run between ISel and MIPeephole.

I hope I explained this ok. Let me know if there are more questions about this.

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

for (MachineInstr *PHI : llvm::reverse(PHIs)) {
     MachineInstrBuilder NewPHI = BuildMI(
         *PHI->getParent(), PHI, PHI->getDebugLoc(), TII->get(PPC::PHI), AccReg);
     for (auto RegMBB : PHIOps)
       NewPHI.add(RegMBB.first).add(RegMBB.second);
     ChangedPHIMap[PHI] = NewPHI.getInstr();
   }
 }
 
+// Check if this instruction is a doubleword swap.
+// Looking for XXPERMDI Result, A, A, 2
+static bool isDoublewordSwap(MachineInstr *PermInstr) {
+  if (PermInstr->getOpcode() != PPC::XXPERMDI)
+    return false;
+
+  MachineOperand &PermInput1 = PermInstr->getOperand(1);
+  MachineOperand &PermInput2 = PermInstr->getOperand(2);
+  unsigned PermType = PermInstr->getOperand(3).getImm();
+  return PermInput1.getReg() == PermInput2.getReg() && PermType == 2;
+}
+
+// Can we prove that this machine operand is a symmetrical vector.
+// Will return true only if it is known that:
+// doubleword[0] == doubleword[1] for Operand.
+static bool isVectorSymmetrical(MachineOperand &VecInput,
+                                MachineRegisterInfo *MRI) {
+  if (!VecInput.isReg())
+    return false;
+
+  MachineInstr *DefVecReg = getVRegDefOrNull(&VecInput, MRI);
+
+  if (!DefVecReg)
+    return false;
+
+  if (!DefVecReg->getDesc().isCommutable())
+    return false;
+
+  MachineOperand &OpInput1 = DefVecReg->getOperand(1);
+  MachineOperand &OpInput2 = DefVecReg->getOperand(2);
+  MachineInstr *Def1 = getVRegDefOrNull(&OpInput1, MRI);
+  MachineInstr *Def2 = getVRegDefOrNull(&OpInput2, MRI);
+
+  if (Def1 && isDoublewordSwap(Def1) &&
+      Def1->getOperand(1).getReg() == OpInput2.getReg())
+    return true;
+
+  if (Def2 && isDoublewordSwap(Def2) &&
+      Def2->getOperand(2).getReg() == OpInput1.getReg())
+    return true;
+
+  return false;
+}
+
 // Perform peephole optimizations.
 bool PPCMIPeephole::simplifyCode() {
   bool Simplified = false;
   bool TrapOpt = false;
   MachineInstr* ToErase = nullptr;
   std::map<MachineInstr *, bool> TOCSaves;
   const TargetRegisterInfo *TRI = &TII->getRegisterInfo();
   NumFunctionsEnteredInMIPeephole++;
for (MachineInstr &MI : MBB) {
           BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(PPC::TRAP));
           TrapOpt = true;
         }
         // We will delete the MI if it will never trap.
         ToErase = &MI;
         Simplified = true;
         break;
       }
+      case PPC::MFVSRLD:
+        // It is more efficient to use MFVSRD instead of MFVSRLD in cases where
+        // it is known that the two doublewords of the vector are identical.
+        MachineOperand &VecInput = MI.getOperand(1);
+        if (!isVectorSymmetrical(VecInput, MRI))
+          break;
+
+        BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(PPC::MFVSRD),
+                MI.getOperand(0).getReg())
+            .addReg(VecInput.getReg(), 0, PPC::sub_64);
+        ToErase = &MI;
+        Simplified = true;
+        break;
       }
     }
 
     // If the last instruction was marked for elimination,
     // remove it now.
     if (ToErase) {
       ToErase->eraseFromParent();
       ToErase = nullptr;

llvm/test/CodeGen/PowerPC/ppc64-mfvsrld-removal.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr \
+; RUN:   -mcpu=pwr9 -mtriple=powerpc64le < %s | FileCheck %s --check-prefix=PWR10LE
+; RUN: llc -verify-machineinstrs -ppc-asm-full-reg-names -ppc-vsr-nums-as-vr \
+; RUN:   -mcpu=pwr9 -mtriple=powerpc64 < %s | FileCheck %s --check-prefix=PWR10BE
+
+; Function Attrs: nounwind
+define dso_local i64 @getVecSplit(i32 noundef signext %a, <2 x i64> noundef %Va, <2 x i64> noundef %Vb) local_unnamed_addr #0 {
+; PWR10LE-LABEL: getVecSplit:
+; PWR10LE:       # %bb.0: # %entry
+; PWR10LE-NEXT:    xxswapd v4, v2
+; PWR10LE-NEXT:    cmplwi r3, 0
+; PWR10LE-NEXT:    vaddudm v2, v4, v2
+; PWR10LE-NEXT:    beq cr0, .LBB0_2
+; PWR10LE-NEXT:  # %bb.1: # %if.then
+; PWR10LE-NEXT:    mfvsrd r3, v2
+; PWR10LE-NEXT:    blr
+; PWR10LE-NEXT:  .LBB0_2: # %if.else
+; PWR10LE-NEXT:    mflr r0
+; PWR10LE-NEXT:    std r0, 16(r1)
+; PWR10LE-NEXT:    stdu r1, -32(r1)
+; PWR10LE-NEXT:    vaddudm v2, v2, v3
+; PWR10LE-NEXT:    bl callee
+; PWR10LE-NEXT:    nop
+; PWR10LE-NEXT:    addi r3, r3, 42
+; PWR10LE-NEXT:    addi r1, r1, 32
+; PWR10LE-NEXT:    ld r0, 16(r1)
+; PWR10LE-NEXT:    mtlr r0
+; PWR10LE-NEXT:    blr
+;
+; PWR10BE-LABEL: getVecSplit:
+; PWR10BE:       # %bb.0: # %entry
+; PWR10BE-NEXT:    xxswapd v4, v2
+; PWR10BE-NEXT:    cmplwi r3, 0
+; PWR10BE-NEXT:    vaddudm v2, v4, v2
+; PWR10BE-NEXT:    beq cr0, .LBB0_2
+; PWR10BE-NEXT:  # %bb.1: # %if.then
+; PWR10BE-NEXT:    mfvsrd r3, v2
+; PWR10BE-NEXT:    blr
+; PWR10BE-NEXT:  .LBB0_2: # %if.else
+; PWR10BE-NEXT:    mflr r0
+; PWR10BE-NEXT:    std r0, 16(r1)
+; PWR10BE-NEXT:    stdu r1, -112(r1)
+; PWR10BE-NEXT:    vaddudm v2, v2, v3
+; PWR10BE-NEXT:    bl callee
+; PWR10BE-NEXT:    nop
+; PWR10BE-NEXT:    addi r3, r3, 42
+; PWR10BE-NEXT:    addi r1, r1, 112
+; PWR10BE-NEXT:    ld r0, 16(r1)
+; PWR10BE-NEXT:    mtlr r0
+; PWR10BE-NEXT:    blr
+entry:
+  %vecins2 = shufflevector <2 x i64> %Va, <2 x i64> undef, <2 x i32> <i32 1, i32 0>
+  %tobool.not = icmp eq i32 %a, 0
+  %add4 = add <2 x i64> %vecins2, %Va
+  br i1 %tobool.not, label %if.else, label %if.then
+
+if.then:                                          ; preds = %entry
+  %vecext3 = extractelement <2 x i64> %add4, i64 0
+  br label %cleanup
+
+if.else:                                          ; preds = %entry
+  %add5 = add <2 x i64> %add4, %Vb
+  %call = tail call i64 @callee(<2 x i64> noundef %add5)
+  %add6 = add nsw i64 %call, 42
+  br label %cleanup
+
+cleanup:                                          ; preds = %if.else, %if.then
+  %retval.0 = phi i64 [ %vecext3, %if.then ], [ %add6, %if.else ]
+  ret i64 %retval.0
+}
+
+declare i64 @callee(<2 x i64> noundef) local_unnamed_addr #0
+
+attributes #0 = { nounwind }

llvm/test/CodeGen/PowerPC/reduce_scalarization.ll

 ; CHECK-P10:       # %bb.0: # %bb
 ; CHECK-P10-NEXT:    plfd f0, Glob1@PCREL+562536(0), 1
 ; CHECK-P10-NEXT:    xxlxor vs1, vs1, vs1
 ; CHECK-P10-NEXT:    xxmrghw vs0, vs0, vs0
 ; CHECK-P10-NEXT:    xvcvspdp vs0, vs0
 ; CHECK-P10-NEXT:    xvcmpeqdp v2, vs1, vs0
 ; CHECK-P10-NEXT:    xxswapd v3, v2
 ; CHECK-P10-NEXT:    xxland vs0, v2, v3
-; CHECK-P10-NEXT:    mfvsrld r3, vs0
+; CHECK-P10-NEXT:    mffprd r3, f0
 ; CHECK-P10-NEXT:    andi. r3, r3, 1
 ; CHECK-P10-NEXT:    bc 4, gt, .LBB5_2
 ; CHECK-P10-NEXT:  # %bb.1: # %bb8
 ; CHECK-P10-NEXT:  .LBB5_2: # %bb7
 ;
 ; CHECK-P10-BE-LABEL: test6:
 ; CHECK-P10-BE:       # %bb.0: # %bb
 ; CHECK-P10-BE-NEXT:    addis r3, r2, Glob1@toc@ha

llvm/test/CodeGen/PowerPC/vector-reduce-add.ll

 ;;
 ;; Vectors of i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.add.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.add.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vaddudm v3, v3, v5
 ; PWR9LE-NEXT:    vaddudm v2, v2, v4
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vaddudm v3, v3, v5
 ; PWR9BE-NEXT:    vaddudm v2, v2, v4
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vaddudm v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vaddudm v3, v3, v5
 ; PWR10LE-NEXT:    vaddudm v2, v2, v4
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vaddudm v3, v3, v5
 ; PWR10BE-NEXT:    vaddudm v2, v2, v4
 ; PWR10BE-NEXT:    vaddudm v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2

 ; PWR9LE-NEXT:    vaddudm v2, v2, v6
 ; PWR9LE-NEXT:    vaddudm v5, v5, v9
 ; PWR9LE-NEXT:    vaddudm v3, v3, v7
 ; PWR9LE-NEXT:    vaddudm v3, v3, v5
 ; PWR9LE-NEXT:    vaddudm v2, v2, v4
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vaddudm v4, v4, v8
 ; PWR9BE-NEXT:    vaddudm v2, v2, v6
 ; PWR9BE-NEXT:    vaddudm v5, v5, v9
 ; PWR9BE-NEXT:    vaddudm v3, v3, v7

 ; PWR10LE-NEXT:    vaddudm v5, v5, v9
 ; PWR10LE-NEXT:    vaddudm v3, v3, v7
 ; PWR10LE-NEXT:    vaddudm v3, v3, v5
 ; PWR10LE-NEXT:    vaddudm v2, v2, v6
 ; PWR10LE-NEXT:    vaddudm v2, v2, v4
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vaddudm v4, v4, v8
 ; PWR10BE-NEXT:    vaddudm v5, v5, v9
 ; PWR10BE-NEXT:    vaddudm v3, v3, v7
 ; PWR10BE-NEXT:    vaddudm v3, v3, v5

 ; PWR9LE-NEXT:    vextsb2d v7, v7
 ; PWR9LE-NEXT:    vperm v2, v2, v2, v8
 ; PWR9LE-NEXT:    vextsb2d v2, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v7
 ; PWR9LE-NEXT:    vaddudm v2, v1, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i64_sign:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    addis r3, r2, .LCPI23_0@toc@ha
 ; PWR9BE-NEXT:    addi r3, r3, .LCPI23_0@toc@l
 ; PWR9BE-NEXT:    lxv v3, 0(r3)
 ; PWR9BE-NEXT:    addis r3, r2, .LCPI23_1@toc@ha

 ; PWR10LE-NEXT:    vaddudm v5, v0, v5
 ; PWR10LE-NEXT:    vaddudm v3, v4, v3
 ; PWR10LE-NEXT:    vaddudm v3, v3, v5
 ; PWR10LE-NEXT:    vaddudm v4, v6, v1
 ; PWR10LE-NEXT:    vaddudm v2, v4, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i64_sign:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    addis r3, r2, .LCPI23_0@toc@ha
 ; PWR10BE-NEXT:    addi r3, r3, .LCPI23_0@toc@l
 ; PWR10BE-NEXT:    lxv v3, 0(r3)
 ; PWR10BE-NEXT:    addis r3, r2, .LCPI23_1@toc@ha

 ; PWR9LE-NEXT:    vperm v8, v4, v2, v8
 ; PWR9LE-NEXT:    vperm v2, v4, v2, v9
 ; PWR9LE-NEXT:    vaddudm v4, v7, v6
 ; PWR9LE-NEXT:    vaddudm v2, v2, v8
 ; PWR9LE-NEXT:    vaddudm v2, v4, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vaddudm v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i8tov16i64_zero:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    addis r3, r2, .LCPI24_0@toc@ha
 ; PWR9BE-NEXT:    xxlxor v4, v4, v4
 ; PWR9BE-NEXT:    addi r3, r3, .LCPI24_0@toc@l
 ; PWR9BE-NEXT:    lxv v3, 0(r3)

 ; PWR10LE-NEXT:    vaddudm v4, v1, v0
 ; PWR10LE-NEXT:    vaddudm v3, v5, v3
 ; PWR10LE-NEXT:    vaddudm v3, v3, v4
 ; PWR10LE-NEXT:    vaddudm v4, v7, v6
 ; PWR10LE-NEXT:    vaddudm v2, v4, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vaddudm v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i8tov16i64_zero:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    addis r3, r2, .LCPI24_0@toc@ha
 ; PWR10BE-NEXT:    xxlxor v4, v4, v4
 ; PWR10BE-NEXT:    addi r3, r3, .LCPI24_0@toc@l
 ; PWR10BE-NEXT:    lxv v3, 0(r3)

llvm/test/CodeGen/PowerPC/vector-reduce-and.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    xxland vs0, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    xxland vs0, v2, v3
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    xxland vs0, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    xxland vs0, v2, v3
 ; PWR10BE-NEXT:    mffprd r3, f0
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.and.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxland vs0, v2, v3
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxland vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxland vs0, v2, v3
 ; PWR9BE-NEXT:    xxswapd v2, vs0
 ; PWR9BE-NEXT:    xxland vs0, vs0, v2
 ; PWR9BE-NEXT:    mffprd r3, f0

 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxland vs0, v3, v5
 ; PWR9LE-NEXT:    xxland vs1, v2, v4
 ; PWR9LE-NEXT:    xxland vs0, vs1, vs0
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxland vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxland vs0, v3, v5
 ; PWR9BE-NEXT:    xxland vs1, v2, v4
 ; PWR9BE-NEXT:    xxland vs0, vs1, vs0
 ; PWR9BE-NEXT:    xxswapd v2, vs0

 ; PWR9LE-NEXT:    xxland vs1, v2, v6
 ; PWR9LE-NEXT:    xxland vs2, v5, v9
 ; PWR9LE-NEXT:    xxland vs3, v3, v7
 ; PWR9LE-NEXT:    xxland vs2, vs3, vs2
 ; PWR9LE-NEXT:    xxland vs0, vs1, vs0
 ; PWR9LE-NEXT:    xxland vs0, vs0, vs2
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxland vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxland vs0, v4, v8
 ; PWR9BE-NEXT:    xxland vs1, v2, v6
 ; PWR9BE-NEXT:    xxland vs2, v5, v9
 ; PWR9BE-NEXT:    xxland vs3, v3, v7

llvm/test/CodeGen/PowerPC/vector-reduce-or.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    xxlor vs0, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    xxlor vs0, v2, v3
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    xxlor vs0, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    xxlor vs0, v2, v3
 ; PWR10BE-NEXT:    mffprd r3, f0
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.or.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxlor vs0, v2, v3
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxlor vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxlor vs0, v2, v3
 ; PWR9BE-NEXT:    xxswapd v2, vs0
 ; PWR9BE-NEXT:    xxlor vs0, vs0, v2
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxlor vs0, v2, v3
 ; PWR10LE-NEXT:    xxswapd v2, vs0
 ; PWR10LE-NEXT:    xxlor vs0, vs0, v2
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxlor vs0, v2, v3
 ; PWR10BE-NEXT:    xxswapd v2, vs0
 ; PWR10BE-NEXT:    xxlor vs0, vs0, v2
 ; PWR10BE-NEXT:    mffprd r3, f0
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.or.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxlor vs0, v3, v5
 ; PWR9LE-NEXT:    xxlor vs1, v2, v4
 ; PWR9LE-NEXT:    xxlor vs0, vs1, vs0
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxlor vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxlor vs0, v3, v5
 ; PWR9BE-NEXT:    xxlor vs1, v2, v4
 ; PWR9BE-NEXT:    xxlor vs0, vs1, vs0
 ; PWR9BE-NEXT:    xxswapd v2, vs0
 ; PWR9BE-NEXT:    xxlor vs0, vs0, v2
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxlor vs0, v3, v5
 ; PWR10LE-NEXT:    xxlor vs1, v2, v4
 ; PWR10LE-NEXT:    xxlor vs0, vs1, vs0
 ; PWR10LE-NEXT:    xxswapd v2, vs0
 ; PWR10LE-NEXT:    xxlor vs0, vs0, v2
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxlor vs0, v3, v5
 ; PWR10BE-NEXT:    xxlor vs1, v2, v4
 ; PWR10BE-NEXT:    xxlor vs0, vs1, vs0
 ; PWR10BE-NEXT:    xxswapd v2, vs0

 ; PWR9LE-NEXT:    xxlor vs1, v2, v6
 ; PWR9LE-NEXT:    xxlor vs2, v5, v9
 ; PWR9LE-NEXT:    xxlor vs3, v3, v7
 ; PWR9LE-NEXT:    xxlor vs2, vs3, vs2
 ; PWR9LE-NEXT:    xxlor vs0, vs1, vs0
 ; PWR9LE-NEXT:    xxlor vs0, vs0, vs2
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxlor vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxlor vs0, v4, v8
 ; PWR9BE-NEXT:    xxlor vs1, v2, v6
 ; PWR9BE-NEXT:    xxlor vs2, v5, v9
 ; PWR9BE-NEXT:    xxlor vs3, v3, v7

 ; PWR10LE-NEXT:    xxlor vs1, v2, v6
 ; PWR10LE-NEXT:    xxlor vs2, v5, v9
 ; PWR10LE-NEXT:    xxlor vs3, v3, v7
 ; PWR10LE-NEXT:    xxlor vs2, vs3, vs2
 ; PWR10LE-NEXT:    xxlor vs0, vs1, vs0
 ; PWR10LE-NEXT:    xxlor vs0, vs0, vs2
 ; PWR10LE-NEXT:    xxswapd v2, vs0
 ; PWR10LE-NEXT:    xxlor vs0, vs0, v2
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxlor vs0, v4, v8
 ; PWR10BE-NEXT:    xxlor vs1, v2, v6
 ; PWR10BE-NEXT:    xxlor vs2, v5, v9
 ; PWR10BE-NEXT:    xxlor vs3, v3, v7

llvm/test/CodeGen/PowerPC/vector-reduce-smax.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.smax.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.smax.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v4
 ; PWR9LE-NEXT:    vmaxsd v3, v3, v5
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v4
 ; PWR9BE-NEXT:    vmaxsd v3, v3, v5
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v4
 ; PWR10LE-NEXT:    vmaxsd v3, v3, v5
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vmaxsd v2, v2, v4
 ; PWR10BE-NEXT:    vmaxsd v3, v3, v5
 ; PWR10BE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2

 ; PWR9LE-NEXT:    vmaxsd v5, v5, v9
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v6
 ; PWR9LE-NEXT:    vmaxsd v4, v4, v8
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v4
 ; PWR9LE-NEXT:    vmaxsd v3, v3, v5
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vmaxsd v3, v3, v7
 ; PWR9BE-NEXT:    vmaxsd v5, v5, v9
 ; PWR9BE-NEXT:    vmaxsd v2, v2, v6
 ; PWR9BE-NEXT:    vmaxsd v4, v4, v8

 ; PWR10LE-NEXT:    vmaxsd v5, v5, v9
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v6
 ; PWR10LE-NEXT:    vmaxsd v4, v4, v8
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v4
 ; PWR10LE-NEXT:    vmaxsd v3, v3, v5
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vmaxsd v3, v3, v7
 ; PWR10BE-NEXT:    vmaxsd v5, v5, v9
 ; PWR10BE-NEXT:    vmaxsd v2, v2, v6
 ; PWR10BE-NEXT:    vmaxsd v4, v4, v8

llvm/test/CodeGen/PowerPC/vector-reduce-smin.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vminsd v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vminsd v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.smin.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vminsd v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vminsd v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vminsd v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vminsd v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.smin.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vminsd v2, v2, v4
 ; PWR9LE-NEXT:    vminsd v3, v3, v5
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vminsd v2, v2, v4
 ; PWR9BE-NEXT:    vminsd v3, v3, v5
 ; PWR9BE-NEXT:    vminsd v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vminsd v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vminsd v2, v2, v4
 ; PWR10LE-NEXT:    vminsd v3, v3, v5
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vminsd v2, v2, v4
 ; PWR10BE-NEXT:    vminsd v3, v3, v5
 ; PWR10BE-NEXT:    vminsd v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2

 ; PWR9LE-NEXT:    vminsd v5, v5, v9
 ; PWR9LE-NEXT:    vminsd v2, v2, v6
 ; PWR9LE-NEXT:    vminsd v4, v4, v8
 ; PWR9LE-NEXT:    vminsd v2, v2, v4
 ; PWR9LE-NEXT:    vminsd v3, v3, v5
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminsd v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vminsd v3, v3, v7
 ; PWR9BE-NEXT:    vminsd v5, v5, v9
 ; PWR9BE-NEXT:    vminsd v2, v2, v6
 ; PWR9BE-NEXT:    vminsd v4, v4, v8

 ; PWR10LE-NEXT:    vminsd v5, v5, v9
 ; PWR10LE-NEXT:    vminsd v2, v2, v6
 ; PWR10LE-NEXT:    vminsd v4, v4, v8
 ; PWR10LE-NEXT:    vminsd v2, v2, v4
 ; PWR10LE-NEXT:    vminsd v3, v3, v5
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminsd v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vminsd v3, v3, v7
 ; PWR10BE-NEXT:    vminsd v5, v5, v9
 ; PWR10BE-NEXT:    vminsd v2, v2, v6
 ; PWR10BE-NEXT:    vminsd v4, v4, v8

llvm/test/CodeGen/PowerPC/vector-reduce-umax.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vmaxud v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vmaxud v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.umax.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vmaxud v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vmaxud v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vmaxud v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vmaxud v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.umax.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vmaxud v2, v2, v4
 ; PWR9LE-NEXT:    vmaxud v3, v3, v5
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vmaxud v2, v2, v4
 ; PWR9BE-NEXT:    vmaxud v3, v3, v5
 ; PWR9BE-NEXT:    vmaxud v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vmaxud v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vmaxud v2, v2, v4
 ; PWR10LE-NEXT:    vmaxud v3, v3, v5
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vmaxud v2, v2, v4
 ; PWR10BE-NEXT:    vmaxud v3, v3, v5
 ; PWR10BE-NEXT:    vmaxud v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2

 ; PWR9LE-NEXT:    vmaxud v5, v5, v9
 ; PWR9LE-NEXT:    vmaxud v2, v2, v6
 ; PWR9LE-NEXT:    vmaxud v4, v4, v8
 ; PWR9LE-NEXT:    vmaxud v2, v2, v4
 ; PWR9LE-NEXT:    vmaxud v3, v3, v5
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vmaxud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vmaxud v3, v3, v7
 ; PWR9BE-NEXT:    vmaxud v5, v5, v9
 ; PWR9BE-NEXT:    vmaxud v2, v2, v6
 ; PWR9BE-NEXT:    vmaxud v4, v4, v8

 ; PWR10LE-NEXT:    vmaxud v5, v5, v9
 ; PWR10LE-NEXT:    vmaxud v2, v2, v6
 ; PWR10LE-NEXT:    vmaxud v4, v4, v8
 ; PWR10LE-NEXT:    vmaxud v2, v2, v4
 ; PWR10LE-NEXT:    vmaxud v3, v3, v5
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vmaxud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vmaxud v3, v3, v7
 ; PWR10BE-NEXT:    vmaxud v5, v5, v9
 ; PWR10BE-NEXT:    vmaxud v2, v2, v6
 ; PWR10BE-NEXT:    vmaxud v4, v4, v8

llvm/test/CodeGen/PowerPC/vector-reduce-umin.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vminud v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vminud v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.umin.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vminud v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vminud v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vminud v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vminud v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vminud v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    vminud v2, v2, v3
 ; PWR10BE-NEXT:    mfvsrd r3, v2
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.umin.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    vminud v2, v2, v4
 ; PWR9LE-NEXT:    vminud v3, v3, v5
 ; PWR9LE-NEXT:    vminud v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vminud v2, v2, v4
 ; PWR9BE-NEXT:    vminud v3, v3, v5
 ; PWR9BE-NEXT:    vminud v2, v2, v3
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    vminud v2, v2, v3
 ; PWR9BE-NEXT:    mfvsrd r3, v2
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    vminud v2, v2, v4
 ; PWR10LE-NEXT:    vminud v3, v3, v5
 ; PWR10LE-NEXT:    vminud v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vminud v2, v2, v4
 ; PWR10BE-NEXT:    vminud v3, v3, v5
 ; PWR10BE-NEXT:    vminud v2, v2, v3
 ; PWR10BE-NEXT:    xxswapd v3, v2

 ; PWR9LE-NEXT:    vminud v5, v5, v9
 ; PWR9LE-NEXT:    vminud v2, v2, v6
 ; PWR9LE-NEXT:    vminud v4, v4, v8
 ; PWR9LE-NEXT:    vminud v2, v2, v4
 ; PWR9LE-NEXT:    vminud v3, v3, v5
 ; PWR9LE-NEXT:    vminud v2, v2, v3
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    vminud v2, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, v2
+; PWR9LE-NEXT:    mfvsrd r3, v2
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    vminud v3, v3, v7
 ; PWR9BE-NEXT:    vminud v5, v5, v9
 ; PWR9BE-NEXT:    vminud v2, v2, v6
 ; PWR9BE-NEXT:    vminud v4, v4, v8

 ; PWR10LE-NEXT:    vminud v5, v5, v9
 ; PWR10LE-NEXT:    vminud v2, v2, v6
 ; PWR10LE-NEXT:    vminud v4, v4, v8
 ; PWR10LE-NEXT:    vminud v2, v2, v4
 ; PWR10LE-NEXT:    vminud v3, v3, v5
 ; PWR10LE-NEXT:    vminud v2, v2, v3
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    vminud v2, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, v2
+; PWR10LE-NEXT:    mfvsrd r3, v2
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    vminud v3, v3, v7
 ; PWR10BE-NEXT:    vminud v5, v5, v9
 ; PWR10BE-NEXT:    vminud v2, v2, v6
 ; PWR10BE-NEXT:    vminud v4, v4, v8

llvm/test/CodeGen/PowerPC/vector-reduce-xor.ll

 ;;
 ;; Vectors of type i64
 ;;
 define dso_local i64 @v2i64(<2 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v2i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxswapd v3, v2
 ; PWR9LE-NEXT:    xxlxor vs0, v2, v3
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v2i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxswapd v3, v2
 ; PWR9BE-NEXT:    xxlxor vs0, v2, v3
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v2i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxswapd v3, v2
 ; PWR10LE-NEXT:    xxlxor vs0, v2, v3
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v2i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxswapd v3, v2
 ; PWR10BE-NEXT:    xxlxor vs0, v2, v3
 ; PWR10BE-NEXT:    mffprd r3, f0
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.xor.v2i64(<2 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v4i64(<4 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v4i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxlxor vs0, v2, v3
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxlxor vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v4i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxlxor vs0, v2, v3
 ; PWR9BE-NEXT:    xxswapd v2, vs0
 ; PWR9BE-NEXT:    xxlxor vs0, vs0, v2
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v4i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxlxor vs0, v2, v3
 ; PWR10LE-NEXT:    xxswapd v2, vs0
 ; PWR10LE-NEXT:    xxlxor vs0, vs0, v2
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v4i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxlxor vs0, v2, v3
 ; PWR10BE-NEXT:    xxswapd v2, vs0
 ; PWR10BE-NEXT:    xxlxor vs0, vs0, v2
 ; PWR10BE-NEXT:    mffprd r3, f0
 ; PWR10BE-NEXT:    blr
 entry:
   %0 = call i64 @llvm.vector.reduce.xor.v4i64(<4 x i64> %a)
   ret i64 %0
 }
 
 define dso_local i64 @v8i64(<8 x i64> %a) local_unnamed_addr #0 {
 ; PWR9LE-LABEL: v8i64:
 ; PWR9LE:       # %bb.0: # %entry
 ; PWR9LE-NEXT:    xxlxor vs0, v3, v5
 ; PWR9LE-NEXT:    xxlxor vs1, v2, v4
 ; PWR9LE-NEXT:    xxlxor vs0, vs1, vs0
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxlxor vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v8i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxlxor vs0, v3, v5
 ; PWR9BE-NEXT:    xxlxor vs1, v2, v4
 ; PWR9BE-NEXT:    xxlxor vs0, vs1, vs0
 ; PWR9BE-NEXT:    xxswapd v2, vs0
 ; PWR9BE-NEXT:    xxlxor vs0, vs0, v2
 ; PWR9BE-NEXT:    mffprd r3, f0
 ; PWR9BE-NEXT:    blr
 ;
 ; PWR10LE-LABEL: v8i64:
 ; PWR10LE:       # %bb.0: # %entry
 ; PWR10LE-NEXT:    xxlxor vs0, v3, v5
 ; PWR10LE-NEXT:    xxlxor vs1, v2, v4
 ; PWR10LE-NEXT:    xxlxor vs0, vs1, vs0
 ; PWR10LE-NEXT:    xxswapd v2, vs0
 ; PWR10LE-NEXT:    xxlxor vs0, vs0, v2
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v8i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxlxor vs0, v3, v5
 ; PWR10BE-NEXT:    xxlxor vs1, v2, v4
 ; PWR10BE-NEXT:    xxlxor vs0, vs1, vs0
 ; PWR10BE-NEXT:    xxswapd v2, vs0

 ; PWR9LE-NEXT:    xxlxor vs1, v2, v6
 ; PWR9LE-NEXT:    xxlxor vs2, v5, v9
 ; PWR9LE-NEXT:    xxlxor vs3, v3, v7
 ; PWR9LE-NEXT:    xxlxor vs2, vs3, vs2
 ; PWR9LE-NEXT:    xxlxor vs0, vs1, vs0
 ; PWR9LE-NEXT:    xxlxor vs0, vs0, vs2
 ; PWR9LE-NEXT:    xxswapd v2, vs0
 ; PWR9LE-NEXT:    xxlxor vs0, vs0, v2
-; PWR9LE-NEXT:    mfvsrld r3, vs0
+; PWR9LE-NEXT:    mffprd r3, f0
 ; PWR9LE-NEXT:    blr
 ;
 ; PWR9BE-LABEL: v16i64:
 ; PWR9BE:       # %bb.0: # %entry
 ; PWR9BE-NEXT:    xxlxor vs0, v4, v8
 ; PWR9BE-NEXT:    xxlxor vs1, v2, v6
 ; PWR9BE-NEXT:    xxlxor vs2, v5, v9
 ; PWR9BE-NEXT:    xxlxor vs3, v3, v7

 ; PWR10LE-NEXT:    xxlxor vs1, v2, v6
 ; PWR10LE-NEXT:    xxlxor vs2, v5, v9
 ; PWR10LE-NEXT:    xxlxor vs3, v3, v7
 ; PWR10LE-NEXT:    xxlxor vs2, vs3, vs2
 ; PWR10LE-NEXT:    xxlxor vs0, vs1, vs0
 ; PWR10LE-NEXT:    xxlxor vs0, vs0, vs2
 ; PWR10LE-NEXT:    xxswapd v2, vs0
 ; PWR10LE-NEXT:    xxlxor vs0, vs0, v2
-; PWR10LE-NEXT:    mfvsrld r3, vs0
+; PWR10LE-NEXT:    mffprd r3, f0
 ; PWR10LE-NEXT:    blr
 ;
 ; PWR10BE-LABEL: v16i64:
 ; PWR10BE:       # %bb.0: # %entry
 ; PWR10BE-NEXT:    xxlxor vs0, v4, v8
 ; PWR10BE-NEXT:    xxlxor vs1, v2, v6
 ; PWR10BE-NEXT:    xxlxor vs2, v5, v9
 ; PWR10BE-NEXT:    xxlxor vs3, v3, v7